Semantic fusion method, computer-readable non-transitory storage medium and system

Abstract

In one embodiment, a computing system accesses a plurality of images captured by one or more cameras from a plurality of camera poses. The computing system generates a plurality of semantic segmentations using the plurality of images, the semantic segmentations comprising semantic information of one or more objects captured in the plurality of images. The computing system accesses a three-dimensional (3D) model of the one or more objects. The computing system uses the plurality of camera poses to determine a corresponding plurality of virtual camera poses relative to the 3D model of the one or more objects. The computing system generates a semantic 3D model by projecting the semantic information of the plurality of semantic segmentations onto the 3D model using the plurality of virtual camera poses.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Application No. 16 / 403,421, filed May 3, 2019, the contents of which are incorporated herein by reference in their entirety for all purposes. Technical Field

[0003] The present disclosure relates generally to controls and interfaces for user interactions and experiences in virtual reality environments.

[0004] background

[0005] Virtual reality is a simulation of a computer-generated environment (e.g., a 3D environment) that a user can interact with in a seemingly real or physical way. A virtual reality system (which can be a single device or a group of devices) can generate the simulation to be displayed to a user on, for example, a virtual reality head-mounted device (headset) or some other display device. The simulation can include images, sounds, tactile feedback, and / or other sensations to simulate a real or imagined environment. As virtual reality becomes more and more prominent, its useful application range is rapidly expanding. The most common applications of virtual reality include games or other interactive content, but other applications (such as viewing of visual media items (e.g., photos, videos) for entertainment or training purposes) are closely behind. The feasibility of using virtual reality to simulate real-life conversations and other user interactions is also being explored.

[0006] Overview of Certain Embodiments

[0007] Various different ways of rendering and interacting with a virtual (or augmented) reality environment are disclosed herein. A virtual reality system can render a virtual environment that can include a virtual space rendered to be displayed to one or more users. Users can view and interact in the virtual space and the broader virtual environment in any suitable manner. In a particular embodiment, the virtual reality system can generate a virtual space to present to the user. As an example and not limitation, the virtual reality system can generate a virtual arena to place the user in it. In a particular embodiment, in order to provide the user with a better understanding of the virtual space, the virtual reality system can label objects within the environment. The label can be a description of the object (e.g., chair, table, etc.), a color coding corresponding to the type of object or what kind of object it is, a bounding box identifying the object, and other types of marking schemes. In addition to providing labels, the system can also generate a mask of the object to be projected into the environment. While such visual representations of various objects provide insight, they also help the virtual reality system by ensuring that the integrity of the environment is maintained. Users can annotate and fix any misidentified objects. For example, if a carpet is mistaken for a floor. The maintenance of object recognition ensures that the virtual reality system can accurately render any environment for the user through the training model. Semantic models can be trained to recognize certain objects and how they are incorporated into a virtual reality environment. For example, multiple objects may correspond to various semantic sets, such as furniture. If a user wants to reconstruct furniture in a virtual reality environment, he or she can quickly remove the furniture for reorganization by using filtering tools. However, filtering tools will not work properly unless objects are correctly identified and grouped into appropriate semantic sets. Other applications may include projecting augmented reality elements in a real-world view. A virtual reality system may have to recognize objects in order to interface appropriately with the real world. For example, a user may want to place a generated table in a real-world view of a restaurant area. The restaurant area may have both carpet and hardwood floors. In order to accurately place the generated table, the virtual reality system must recognize separate objects for the floor and carpet. This can be achieved by building a semantic model that correctly identifies objects within a scene by generating multiple segmentations that provide information corresponding to the identified objects.

[0008] In order to generate a 3D semantic model, in a particular embodiment, the virtual reality system may access multiple images captured by one or more cameras from multiple camera poses. As an example and not limitation, the image may be captured from a real-world environment (such as an office environment). As another example and not limitation, the image may be generated or accessed from an image storage. In a particular embodiment, the virtual reality system may use the image to generate multiple semantic segmentations, which include semantic information of the object captured within the image. Although semantic segmentation and semantic information are discussed, other types of segmentation and information may also be generated using the image. In a particular embodiment, the virtual reality system may access a three-dimensional (3D) model of the object. As an example and not limitation, the virtual reality system may generate a 3D model of the object based on the captured image by rendering the 3D model. In a particular embodiment, the virtual reality system may use the camera pose to determine a corresponding multiple virtual camera poses relative to the 3D model of the object. In a particular embodiment, the virtual reality system may generate a semantic 3D model by projecting the semantic information of multiple semantic segmentations onto the 3D model using multiple virtual camera poses.

[0009] Various different ways of rendering and interacting with a virtual (or augmented) reality environment are disclosed herein. A virtual reality system can render a virtual environment, which can include a virtual space that is rendered for display to one or more users. Users can view and interact with the virtual space and the broader virtual environment in any suitable manner. One goal of the disclosed methods is to provide an intuitive experience for users - giving users a sense of "presence", or the feeling that they are actually in the virtual environment.

[0010] Embodiments of the present invention may include an artificial reality system or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some way before being presented to a user, and may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or generated content combined with captured content (e.g., photos of the real world). Artificial reality content may include video, audio, tactile feedback, or some combination thereof, and any of which may be presented in a single channel or in multiple channels (e.g., stereoscopic video that produces a three-dimensional effect to the audience). In addition, in some embodiments, artificial reality may be associated with an application, product, accessory, service, or some combination thereof, which is used, for example, to create content in artificial reality and / or to be used in artificial reality (e.g., to perform activities in artificial reality). Artificial reality systems that provide artificial reality content can be implemented on a variety of platforms, including a head-mounted display (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0011] The embodiments disclosed herein are merely examples, and the scope of the present disclosure is not limited to them. Specific embodiments may include all, some or none of the components, elements, features, functions, operations or steps of the embodiments disclosed above. Embodiments according to the present invention are specifically disclosed in the attached claims, relating to methods, storage media, systems and computer program products, wherein any feature mentioned in one claim category (e.g., method) may also be claimed in another claim category (e.g., system). The dependencies or back references in the attached claims are selected only for formal reasons. However, any subject matter generated by intentional back references to any previous claim (especially multiple dependencies) may also be claimed, so that any combination of claims and their features is disclosed and may be claimed, regardless of the dependencies selected in the attached claims. The subject matter that may be claimed includes not only the combination of features as set forth in the attached claims, but also any other combination of features in the claims, wherein each feature mentioned in the claims may be combined with any other feature or combination of other features in the claims. Furthermore, any of the embodiments and features described or depicted herein may be claimed in a separate claim and / or in any combination with any embodiment or feature described or depicted herein or in any combination with any features of the appended claims.

[0012] In one aspect, the present invention discloses a method according to the attached claims. The method comprises, by a computing system:

[0013] accessing a plurality of images captured by one or more cameras from a plurality of camera poses;

[0014] generating, using the plurality of images, a plurality of semantic segmentations comprising semantic information of one or more objects captured in the plurality of images;

[0015] accessing a three-dimensional (3D) model of one or more objects;

[0016] determining a corresponding plurality of virtual camera poses relative to the 3D model of the one or more objects using the plurality of camera poses;

[0017] A semantic 3D model is generated by projecting the semantic information of multiple semantic segmentations to the 3D model using multiple virtual camera poses.

[0018] In an embodiment of the present invention, the method may further include generating a plurality of geometry-based segmentations using the plurality of images, the segmentations including geometric information of one or more objects captured in the plurality of images. Optionally, generating a semantic 3D model by projecting semantic information of the plurality of semantic segmentations may further include projecting semantic information corresponding to the one or more objects captured in the plurality of images using the geometric information of the one or more objects.

[0019] In another embodiment of the present invention, the method may further include generating multiple instance segmentations using multiple images, the instance segmentations including object identifications of one or more objects captured in the multiple images. Optionally, the method may further include generating an instance 3D model by projecting the object identifications of the multiple instance segmentations onto the 3D model using multiple virtual camera poses, wherein generating the instance 3D model may further include combining the object identifications from each of the multiple instance segmentations to apply to one of the one or more objects.

[0020] In another embodiment of the present invention, the method may further include generating a semantic tree based on semantic information of one or more objects captured in the plurality of images, wherein the semantic tree may include a plurality of semantic levels, and wherein each semantic level may correspond to a threshold amount of semantic information. Optionally, the plurality of semantic levels may further include one or more of the following:

[0021] A first semantic level, which includes information corresponding to each of the one or more objects,

[0022] a second semantic level, which includes information corresponding to object classes of one or more objects, and

[0023] The third semantic level includes information corresponding to a semantic set of one or more objects.

[0024] In another embodiment of the present invention, the method may further include generating a semantic tree based on semantic information of one or more objects captured in multiple images, wherein the semantic tree may include multiple semantic levels, wherein each semantic level may correspond to a threshold amount of semantic information, and wherein projecting the semantic information may further include projecting semantic information of a specific semantic level onto the 3D model.

[0025] In another embodiment of the present invention, in the method, a 3D model of one or more objects may be generated based on a plurality of images.

[0026] In another embodiment of the present invention, the method may further include calculating one or more confidence values ​​that each semantic information of one or more objects from the plurality of semantic segmentations corresponds to a corresponding object. Optionally, the object may be associated with a plurality of environments, and one or more of the confidence values ​​may be based on whether the semantic information of the one or more objects corresponds to an object in a particular environment.

[0027] In another embodiment of the present invention, the method may further include calculating one or more confidence values ​​that each semantic information of one or more objects from the multiple semantic segmentations corresponds to the corresponding object. Optionally, projecting the semantic information of the multiple semantic segmentations to the 3D model may further include: determining that the confidence value of the semantic information of an object exceeds a threshold confidence value; and in response to the confidence value exceeding the threshold confidence value, projecting the semantic information to the 3D model of the corresponding object. Alternatively or in addition, discarding the semantic information associated with the confidence value less than the threshold confidence value.

[0028] In another embodiment of the present invention, in the method, generating the semantic 3D model may include combining semantic information from each of the plurality of semantic segmentations to apply to one of the one or more objects.

[0029] In another embodiment of the present invention, in the method, projecting the semantic information to the 3D model may include adding tags corresponding to corresponding objects in the 3D model.

[0030] In one aspect, the invention is directed to one or more computer-readable non-transitory storage media containing software that, when executed, is operable to perform the above-described method.

[0031] In one aspect, the present invention also relates to a system comprising one or more processors and a non-transitory memory coupled to the processor, the non-transitory memory comprising instructions executable by the processor, the processor being operable to perform the above method when executing the instructions. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The patent or application file contains at least one drawing executed in color. Copies of the patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0034] Figure 1A-Figure 1C An example process of performing segmentation for use in a semantic fusion algorithm is shown.

[0035] Figure 2A and Figure 2B An example flow chart illustrating a semantic fusion process.

[0036] Figure 3A-3C Another example process of performing segmentation for use in a semantic fusion algorithm is shown.

[0037] Figure 4A and Figure 4B An example projection process of semantic information is shown.

[0038] Figure 5 An example process of projecting information from separate segmentations to an object is shown.

[0039] Figure 6 An example semantic tree and different components of the semantic tree are shown.

[0040] Figure 7 An example network environment associated with a virtual reality system is shown.

[0041] Figure 8 An example method for generating a semantic model is shown.

[0042] Fig. 9 An example computer system is shown.

[0043] Description of Example Embodiments

[0044] Semantic scene understanding can be a fundamental problem in computer vision, robotics, and graphics. To facilitate research, large-scale, high-quality ground truth datasets are crucial for data-hungry algorithms such as deep learning. While large-scale semantic RGB-D datasets have an impressive scale, little attention has been paid to how to efficiently obtain detailed annotations that also maintain consistency between multi-view 2D images and 3D reconstructions. Therefore, a tool may be needed to enable humans to annotate directly on 3D meshes and clean up the mesh geometry to enhance semantic propagation between 2D and 3D. For large-scale ground truth collections, a closed-loop workflow that learns from existing semantic segmentation to guide future annotations may contribute to these capabilities.

[0045] Existing annotations may often be contaminated by labeling noise, correspondence between 2D and 3D labels may be non-existent or inconsistent, and detailed annotations that respect object boundaries may be difficult to obtain. In addition, 3D datasets may rely heavily on structured light depth sensors for dense reconstruction. Missing surfaces may be common during reconstruction due to occlusion and lack of coverage as well as sensor limitations. This fact can lead to incorrect data association when projecting 3D into 2D image space and vice versa. This may severely limit the use of semantic 3D datasets to train and test 2D machine learning algorithms. For large-scale annotations, annotation efficiency may be an important concern. Along these lines, existing annotations can help further labeling. The workflow described in this paper can provide a segmentation-assisted free-form mesh labeling algorithm that produces hierarchical annotations with better details and efficiency; a human-assisted geometry correction technique that inserts missing surfaces to enhance 2D / 3D association; a closed-loop guided annotation scheme that trains instance semantic annotations based on annotated data and integrates the predictions back into the mesh segmentation.

[0046] In certain embodiments, rendering errors can be reduced and better ground truth annotations can be produced by using reconstructions corrected using the methods described herein. Our annotated model can render 2D label images with approximately 20% higher accuracy than state-of-the-art segmentation-based algorithms. In certain embodiments, multiple images can be captured and used to learn semantic instance predictions in these images using a segmentation algorithm. As an example and not limitation, Mask-RCNN can be used to learn the process of semantic instance prediction.

[0047] refer to Figure 1A, showing the first part of the semantic fusion process 100. In certain embodiments, the system may perform the semantic fusion process 100 described herein. As an example and not limitation, the virtual reality system may perform the semantic fusion process 100. Process 100 begins with a camera view at a specific camera pose shown in image 102. In certain embodiments, the system may perform a segmentation process on image 102. As an example and not limitation, the system may use the Mask-RCNN algorithm to identify objects within image 102 and generate information corresponding to objects within the first image 102. In certain embodiments, the various segmentations that the system may perform may include geometry-based segmentation, instance segmentation, and semantic segmentation. In certain embodiments, geometry-based segmentation may identify several objects within image 102 and geometric information of these objects. In certain embodiments, instance segmentation may identify instances of different objects and assign object identities to each of these objects. These assignments of object identities may be soft assignments, which may be determined in further processes. For example, in image 102, separate instances of different objects identified may be marked with different colors. The predictions of different objects may not always be correct. For example, a chair with an orange portion and a yellow-green portion is identified in image 102. These portions may be masks applied to image 102 as predictions of individual instances of objects within image 102. As a result of some incorrect recognitions, semantic fusion process 100 aggregates information from multiple images to correctly recognize objects in the environment. In certain embodiments, semantic segmentation may generate semantic information for each recognized object within image 102. As an example and not limitation, a chair may be recognized and a green mask applied thereto identifies the object as a chair. In certain embodiments, the system may project the information collected from the segmentation onto a 3D model of the environment and / or object. The projection may be a mask applied to the 3D model. In certain embodiments, the 3D model of the environment and / or object may be generated based on multiple images. The system may project instance-level segmentation information collected from image 102 onto the 3D model, as shown in image 104. Instance-level segmentation information may not provide a complete picture of what the environment is, but only what is captured in image 102. As such, for example, since a chair is recognized as two different objects, instance-level segmentation information may show different instances of an object in different colors, as shown in image 104. Similarly, semantic level segmentation information can be projected to the 3D model, as shown in image 106. By correctly identifying the objects, the system can correctly label the objects within the 3D model based on the content captured and generated from image 102. As shown in image 106, since only partial information is collected, only information related to the content captured in image 102 can be projected to the 3D model, as shown in image 106.

[0048] refer to Figure 1B, showing an example result of the semantic fusion process 100. In a particular embodiment, the system may access multiple images captured by one or more cameras from multiple camera poses to generate multiple segmentations including information corresponding to objects within the images. In a particular embodiment, the system may combine the information to project the information from multiple segmentations onto a 3D model. A 3D model scene of an office may be captured in image 108a, where the system has accessed images corresponding to the scene in multiple camera poses to generate segmentations. Information collected from multiple segmentations can provide a more complete picture by understanding different objects in the environment. In a particular embodiment, the system may project instance-level segmentation information onto a 3D model in image 110a, which appears more complete than image 102 because it combines information collected from multiple images. For example, a more complete shape, such as a green chair or a blue chair, is shown. Similarly, in a particular embodiment, the system may project semantic-level segmentation information onto a 3D model in image 112. As an example and not limitation, all chairs may be projected with a green mask to indicate that they have the same semantic meaning (e.g., they are chairs). In a particular embodiment, the system may use multiple camera poses of the captured image to determine corresponding multiple virtual camera poses relative to the 3D model. In a particular embodiment, the virtual camera poses may be used to project information from the segmentation onto the 3D model. As an example and not limitation, the system may generate a semantic 3D model by projecting semantic information of multiple semantic segmentations onto the 3D model using the virtual camera poses shown in image 112. As another example and not limitation, the system may generate an instance 3D model by projecting object identifications of multiple instance segmentations onto the 3D model using multiple virtual camera poses as shown in 110a. In a particular embodiment, generating an instance 3D model may include combining object identifications from each instance segmentation to be applied to one or more objects in the 3D model.

[0049] refer to Figure 1C , shows another example result of the semantic fusion process 100. In a specific embodiment, similar to Figure 1B, the system may access multiple images captured by one or more cameras from multiple camera poses to generate multiple segmentations including information corresponding to objects within the images. Another 3D model scene of an office may be captured in image 108b, where the system has accessed images corresponding to the scene in multiple camera poses to generate segmentations. In a particular embodiment, the 3D model scene may be rendered based on multiple captured images of the real-world environment. In a particular embodiment, the system may project instance-level segmentation information onto the 3D model in image 110b. In a particular embodiment, the system may project semantic-level segmentation information onto the 3D model in image 114. Semantic information may also include a bounding box that also corresponds to an object. In a particular embodiment, the bounding box may be a tighter fit relative to the object. In a particular embodiment, the orientation of the bounding box may be adjusted to be flush with the floor of the 3D model scene. In a particular embodiment, the system may identify the floor, walls, and ceiling as separate objects.

[0050] Figure 2A and Figure 2B An example flow chart 200 for generating large-scale semantic annotations in 3D is shown. Figure 2A In certain embodiments, first, the system may access a 3D mesh for 3D mesh reconstruction 202, and may compute a geometry-based segmentation 204 to assist in annotation. In certain embodiments, initially, in 3D mesh annotation 206, the annotator may then refine and semantically annotate the pre-segmentation using free-form mesh drawing. In certain embodiments, the annotation may be done by the system or a human annotator. In certain embodiments, the process 200a may continue to model completion 208. The annotated mesh may then be rendered to obtain a fully annotated 2D sequence in 2D label generation 210, which may be used to train a deep learning algorithm for semantic understanding in semantic learning 212. In certain embodiments, assuming sufficient human segmentation and annotation of the mesh, the loop may be closed by using the trained model to infer semantics in image space and fusing these predictions into a consistent segmentation on the 3D mesh using a learned-based segmentation 214.

[0051] refer to Figure 2B , after subsequently training the model, the system can utilize process 200b to generate large-scale 3D annotations. In certain embodiments, the system can access the 3D mesh for 3D mesh reconstruction 202, and can compute geometry-based segmentation 204 to assist in annotation. In certain embodiments, the system can perform learning-based segmentation 214 and then perform 3D mesh annotation based on information collected from learning-based segmentation 214. In certain embodiments, the system can perform model completion 208 after annotating the 3D mesh. In certain embodiments, the system can continue to perform process 200a after training the model to continue training the model.

[0052] refer to Figure 3A-3C , an example semantic segmentation process is shown to perform a semantic fusion process. Figure 3A An example segmentation process is shown. In particular embodiments, a system may access an image 300a of an object 304. In particular embodiments, object 304 may be part of a rendered 3D scene, a real-world scene, etc. In particular embodiments, a system may perform semantic segmentation on image 300a and project semantic level segmentation information onto object 304, as shown in image 302a. In particular embodiments, a system may apply grids 306a and 308a to objects 304a and 304b, respectively. In particular embodiments, grids 306a and 308a may provide semantic meaning corresponding to their respective objects 304. By way of example and not limitation, grid 306a may indicate that object 304a is a chair. Reference Figure 3B , the system may access another image 300b of object 304. Image 300b may be from another camera pose of the same object 304. In certain embodiments, the system may perform semantic segmentation on image 300b and project the semantic level segmentation information onto object 304, as shown in image 302b. In certain embodiments, the system may apply meshes 306b and 308b to object 304. In certain embodiments, the system may perform the process of accessing an image and performing segmentation on the image to generate an annotated 3D model. By way of example and not limitation, the system may perform semantic segmentation on multiple images and generate a semantic 3D model based on the multiple semantic segmentations. Reference Figure 3C , the system can apply a 3D mesh 312 to the chair, as shown in image 310. In a particular embodiment, a semantic 3D model can be generated based on a combination of semantic information from the semantic segmentations performed in images 302a, 302b. When in a first camera pose, the front of the chair is seen and labeled as a chair, a second camera pose may identify the back of the chair as a chair. Typically, it is difficult or impossible to see both the front and back of a chair in one image. As such, in the 3D model, the system can appropriately generate a semantic 3D model by combining semantic information collected from separate segmentations.

[0053] refer to Figure 4A and Figure 4B , an example process 400 for projecting information onto an object within a 3D scene. Figure 4A , capturing an object 402 in a 3D scene with a camera 404. The camera 404 may be in a particular camera pose for viewing the object 402 within the 3D scene. Figure 4B, camera 404 can project information onto object 402 via rays 406 projected from camera 404 at a particular camera pose. In certain embodiments, the system can perform segmentation to generate information to be projected onto object 402. In certain embodiments, the system can compile semantic information, object identification, and geometric information related to object 402. In certain embodiments, the system can use camera 404 to project information onto object 402 via rays 406. In certain embodiments, rays 406 can include information related to the object. As an example and not limitation, rays 406 can indicate that object 402 is a cube.

[0054] refer to Figure 5 , showing another process of generating an annotated 3D model. In a particular embodiment, image 502 shows a chair that has information from several segmentations projected onto it. In a particular embodiment, the system can perform the segmentation and project the information onto the chair shown in different colors. For example, the yellow portion can correspond to one segmentation and the red portion can correspond to another segmentation. The system can project the information from the different segmentations and compile the information into a single mask / mesh, as shown in image 504. In a particular embodiment, the system can generate a red mask to be applied to the chair in image 504. The red mask can include information indicating that the object is a chair.

[0055] In a specific embodiment, the 3D color grid M can be composed of a set of vertices V = {vi∈R 3}, edges E = {eij} and polygon primitives P = {pi}. The attributes of vertex vi can include the unit normal n i ∈R 3 and color c i ∈R 3 The mesh M can be divided into disjoint segments Mi and assigned labels l k In certain embodiments, instance and semantic partitions may be represented by M s i and M q i In certain embodiments, the partitions can be generated from the segmentation process. An RGB-D record can be a sequence {SI of color and depth images t ∈R3,D t ∈R,ξ t ∈R 6 In certain embodiments, a 6DoF rigid body motion ξ may be assumed for each frame. t Given a camera model, the projection of a 3D point v onto a 2D image can be represented by x = π(v), and x∈R2, the back projection is represented by v = π -1(x) represents. To calculate the color distance, the CIELab color space can be used instead of RGB, and Δ(ci, cj) can represent the Euclidean color distance in the CIELab space.

[0056] In a particular embodiment, annotation can be a process where a human interacts with a 3D mesh. To calculate where the annotator clicked on the mesh, OpenGL can be used to render the polygon index into a frame buffer and read out the index under the mouse position. This solution may be more efficient than ray tracing. Similarly, to obtain a 2D label from an annotated mesh, the label value can be looked up by the rendered primitive index.

[0057] In certain embodiments, the annotation algorithm can be used to generate accurate semantic instance labels on the 3D mesh and generate 2D renderings therefrom. In certain embodiments, the benefit of the annotation algorithm can be to help humans efficiently and accurately label semantic objects, ensure labeling consistency between observations from different viewpoints, and maintain close association between 3D reconstruction and 2D images.

[0058] To this end, the proposed tool can be run on a 3D colored mesh. The algorithm can be initialized with a geometric segmentation for annotation. Annotation can be an iterative process between connected segments and varying segments in order to correctly capture details through free-form drawing along the mesh manifold. View rendering can be used to propagate the annotation results from 3D to 2D. To compensate for rendering errors due to imperfect reconstruction, human-assisted geometry completion can be used.

[0059] In a particular embodiment, the process of annotation may be to create a mapping from primitives to tag domains: i →lj. When the mapping f is one-to-one, organizing the results may be trivial. However, given how annotated models are used in practice, it may be desirable to support multi-valued mappings. By way of example and not limitation, the same primitive may be a carpet for one application, and a floor for another. Similarly, hierarchical relationships may be the basis for semantic relationships. By way of example and not limitation, almost any object can be decomposed into smaller parts or grouped into new categories. To support one-to-many mappings and hierarchical relationships in segmentation, a semantic tree may be generated, such as Figure 6 shown.

[0060] refer to Figure 6, the semantic tree 600 is shown to include several levels 602. In a particular embodiment, the leaf nodes may contain sets of primitives corresponding to segments generated by a pre-segmentation algorithm or free-form drawing. These leaves can be connected into a tree to represent different semantics. In a particular embodiment, the hierarchy can be defined as follows: level 0 602a represents the aforementioned non-semantic leaf segmentation, level 1 602b represents object instances, level 2 602c represents object classes, and level 3 602d and above encode higher-level semantic sets. With this definition, trees rendered at different levels naturally produce annotations of different aspects. This is in Figure 6 As shown, until level 3 of the semantic tree. In a particular embodiment, each instance of object 606 may include one or more non-semantic leaf nodes 604. As an example and not limitation, each cushion 606, sofa 606 and table 606 may include a non-semantic leaf node 604. In a particular embodiment, object class 608 may include one or more individual instances of object 606. As an example and not limitation, multiple cushions 606 constitute object class 608 "cushions". That is, in a particular embodiment, all objects of the same class constitute object class 608. In a particular embodiment, semantic set 610 may include several object classes 608 and individual objects 606. As an example and not limitation, semantic set 610 "furniture" may include object class 608 "cushions" and all other object instances 606. Object 606 labeled as "furniture" is included in the "furniture" semantic set 610. In a particular embodiment, semantic tree 600 may be generated based on semantic information accumulated from semantic segmentation performed on multiple images related to a 3D scene. In certain embodiments, each semantic level 602 includes a threshold amount of semantic information. By way of example and not limitation, each object instance 606 in level 1 602b includes semantic information associated with the respective object. Additionally, the next level 2 602c includes the semantic information of the lower level 602 and additional semantic information. In certain embodiments, the system may project semantic information corresponding to a particular level 602 of the semantic tree 600 to the 3D model. By way of example and not limitation, the system may project semantic information corresponding to level 3 602d indicating that several objects are part of the semantic set 610 "furniture".

[0061] In certain embodiments, to extract a reasonable initial segmentation, a robust plane segmentation algorithm may be used that: (1) runs directional segmentation through the DP-vMF mean algorithm, and (2) runs a connected component analysis on each directional segment of the manifold along the grid. In certain embodiments, the eigenvalues ​​λ of the covariance matrix of all points in the segment are analyzed. 1 <λ 2 <λ 3 , the segment can be classified as planar. In a particular embodiment, if λ 1 <<λ 2And 2 <<λ 3 , then the segment can be considered planar. Any segment of the mesh that is not classified as planar can be refined by the Felsenszwalb segmentation algorithm. This may lead to a finer segmentation of all non-planar fragments, which is more useful for the next step of free-form segmentation adjustment and semantic annotation.

[0062] In a particular embodiment, to further clean up the resulting segmentation, a 3D bilateral filter may be run as the final step of pre-segmentation. In a particular embodiment, the bilateral filter may work along the manifold of the mesh using the following weighting function:

[0063]

[0064] In many cases, connecting segments may not produce the desired annotations. This may be because 1) the boundaries of the segments often do not coincide with the actual objects, 2) some segments connect multiple object parts, and 3) the heuristic pre-segmentation algorithm cannot distinguish between objects, for example, separating carpet and floor. In certain embodiments, to address this issue, SceneNN can produce segmentations and allow the annotator to break up large segments by switching from coarse segmentation to fine segmentation. Therefore, free-form drawing along the grid manifold can be used to refine the segments. Based on the seed primitives selected by the user, Algorithm 1 can describe the drawing method. Algorithm 1 can use region growing along the grid to locate the segments. In certain embodiments, the region growing can be adjusted using three parameters. Parameter a t The smoothness can be adjusted by comparing the normal direction, parameter d t The Euclidean distance to the seed primitive can be restricted, parameter S t Color similarity can be adjusted. With the appropriate combination of these parameters, region growing can flexibly select any surface patch, from large planar structures to small curved areas. It can also be able to extract detailed texture patterns, such as posters on a wall.

[0065]

[0066] In certain embodiments, free-form drawing can introduce topological changes to the semantic tree. In order to reflect these changes and maintain the tree structure, affected tree branches can be detected and their empty twin branches can be created. Leaves affected by the drawing may be split into two. Unselected primitives remain unchanged, and selected primitives can be extracted and transferred to twin branches.

[0067] In certain embodiments, many indoor reconstructions may rely on structured light sensors to obtain dense reconstructions. Due to sensor limitations and imperfect capture coverage, the resulting mesh may often contain missing surfaces. This can result in incorrect projections, with missing surfaces behind the reconstruction being projected onto the 2D image. Comparing the rendered depth to the original depth image may detect some errors, however, depth may not always be available. In order to maintain a close connection between the 3D annotations and the 2D image, two simple yet effective techniques can be used to repair the reconstruction.

[0068] In certain embodiments, the missing surface may be generally planar, so the annotator may mark a polygon B around the planar hole. i Then the least squares plane fitting can be used to estimate the i :=(n i ,d) parameterized plane equation,

[0069]

[0070] To stitch the mesh, the planar patches can be directly integrated into the SDF volume used to reconstruct the model. To update the voxels around the target hole, the SDF value F(v) can be calculated as the distance between the voxel center v and the voxel at the target plane

[0071]

[0072] Once the SDF values ​​are updated, the mesh can be regenerated using the standard marching cubes algorithm. The augmented polygons can be automatically assigned to the most likely boundary primitive labels.

[0073] In certain embodiments, in order to complete the missing surface of a cylindrical structure (e.g., a partially reconstructed column), a technique can be implemented to complete the missing surface of the cylindrical structure. The idea may be to first estimate the curve of the central axis and the cross-sectional shape of the object, and then complete the missing surface by scanning the cross-sectional curve along the central axis. In certain embodiments, straight dominant support structures may lead to rendering errors. Instead of estimating the L1 medial axis to approximate the central axis, the central axis of the object can be estimated. First, the main principal components can be calculated by PCA to approximate the central axis. The adjacent vertices are then projected onto the tangent plane, and the cross-sectional shape can be estimated by fitting non-uniform rational basis splines (NURBS). Finally, the axis can be adjusted to the center of the cross-sectional NURBS. In addition, the scanning direction can be adjusted by the annotator to obtain the best results.

[0074] In certain embodiments, these two techniques may not optimally correct reconstruction errors. However, they can effectively reduce the most common rendering errors and produce better label propagation from 3D models to 2D images.

[0075] In certain embodiments, annotation can be an expensive process, and therefore, incorporating machine intelligence to assist human operators is important. In certain embodiments, a closed-loop annotation scheme can be used to guide large-scale semantic annotation collection. In order to propagate understanding from multi-view 2D images to obtain consistent predictions of 3D meshes, a method of fusing multi-view semantic instances onto the 3D mesh can be used. In addition, a technique can be used to bridge the gap between vision-based and geometry-based segmentation.

[0076] In a particular embodiment, the first step in closing the annotation loop can be to train a machine learning algorithm. To train a 2D convolutional neural network, a 3D semantic mesh can be rendered into the corresponding camera view to obtain a densely annotated video sequence. Due to noise and errors in pose estimation, model reconstruction, and camera calibration, the rendered label image may not always fully respect the object boundaries in the original color image. Missing labels in the mesh annotation and missing surfaces in the reconstruction may also cause noise in the label rendering. To correct these artifacts, a 2D joint bilateral filter can be applied to smooth the rendering while preserving edges. Similar to the 3D bilateral filter,

[0077]

[0078] where σ c 2 , σ x 2 and σ v 2 Can be color, 2D pixel coordinates, and 3D Euclidean variance. The weight parameter γ j can be an estimate of the certainty of the rendered labels. Rendered labels may be more prone to errors around object boundaries and label jumps. Therefore, the combined edge map can be estimated as the gradient of the strong image plus the label boundaries. The weighted map γ can then be calculated as the distance to the nearest edge point and then normalized to the range [0,1]. The values ​​γi can also be used to calculate the dynamic filter window. Assuming the maximum window size is K, the filter size for pixel i can be given by γjF.

[0079] In a specific embodiment, a method for computing instance-level semantic segmentation in 3D may be developed as follows. More specifically, the method may be to calculate for each primitive p on the mesh M i Find the tuple of semantic and instance id (M s i ,M q i). As shown in Algorithm 2, this method can be run in three main stages. First, the Mask-RCNN model can be used to detect each image frame I in the original video capture sequence S. t In particular, Mask-RCNN can find a list of object masks {dk}, each of which can be associated with a semantic class ck and a confidence score αk. Using the projection function discovered during 3D reconstruction, each 2D detection can be mapped onto a 3D mesh (lines 8-10). Each mapped detection can be represented by the same semantic class c k , a set of face ids m on the 3D mesh k And the prediction score of Mask-RCNN a k The corresponding confidence score r is combined with the geometric structure information of the mesh (i.e., the camera pose relative to the mesh) k In the second stage, the set of individual detections D can be fused together to obtain a semantic level segmentation s, as shown in line 10. In the third stage, since each instance may be restricted to have one semantic id, it is possible to generate a semantic segmentation s for each semantic class s. k Instance-level segmentation is performed separately, as shown in lines 12-16. The details of the latter two stages are presented as follows.

[0080]

[0081] In a particular embodiment, semantic fusion can find a partition s of the mesh by aggregating the individual detections in D as outlined in Algorithm 3. Specifically, the 3D volume m of the instance can be computed in Lines 6-8. t k (a collection of face IDs on a grid) accumulate the confidence count r for each detection t k , and the partition can be obtained in line 9 by assigning each face id to the semantic class with the most counts. In a specific embodiment, the clustering process can be repeated several times to remove detection instances that are inconsistent with the current partition (line 7), which may be important for removing false-positive detections by Mask-RCNN. In this work, the detection volume m t k Whether more than half of S The predicted semantic class c in t k to determine consistency.

[0082] In certain embodiments, instance-level segmentation can be more difficult because instance ids between different observations cannot be directly associated. For example, a chair in image A may correspond to any chair in image B or not. More importantly, since the detections are initially derived from 2D images, different detections of the same object instance may correspond to images from different viewpoints and thus have little overlap on the 3D grid. To address this challenge, Algorithm 4 can be used to classify the instance ids of each semantic class s given a s. k A set of detection D sk When finding instance-level partition M q sk .

[0083] The proposed method can work similarly to expectation maximization. In a specific embodiment, the algorithm can iterate between the following steps: (i) find a soft assignment z for each detection t (lines 9-14) and (ii) aggregate the confidence counts of the assigned detections {u h} to update instance partition M q sk (Lines 15-19). In this work, the intersection over union (IoU) of detections may be computed with each existing partition (Lines 11-13). In a particular embodiment, given the limited field of view of the camera, the intersection computation between detections and instance partitions may be restricted to the portion of the grid visible to the corresponding image of the detection. A key difference from the classical clustering problem here may be that an observation t may have multiple associated detections. As an example and not limitation, Mark-RCNN may find multiple instances of the same semantic class in an image. As another example and not limitation, two detections from the same image may correspond to two separate instances, so the soft assignment step in Line 14 needs to consider the IoU between each pair of detections and instance partitions. In addition, for semantic classes with large physical size instances (such as doors and walls), each detection typically only obtains a small partial view of the object instance. In a particular embodiment, the IoU matrix may also be used as a metric to determine whether multiple partitions should be merged into a single object (Lines 17-18).

[0084]

[0085]

[0086] In certain embodiments, in the early stages of data collection, the available annotated data may be quite limited. The lack of annotated data may result in the inability to always make correct predictions for each mesh primitive based on the semantic and instance segmentation fused by Mask-RCNN. As an example and not limitation, half of the meshes may not be predicted. However, by using a conservative fusion strategy, most of the valid predictions can be correct. In certain embodiments, geometry-based segmentation can provide concise and useful predictions about object classes. As an example and not limitation, structural surfaces (such as floors and walls) can be well segmented. In certain embodiments, semantic predictions can be conditioned on geometric segments. That is, in certain embodiments, generating an annotated 3D model by projecting information from the segmentation may include using the geometric information of the object to project additional information corresponding to the object in the 3D model. As an example and not limitation, generating a semantic 3D model by projecting semantic information of multiple semantic segmentations may include using the geometric information of the object to project semantic information corresponding to the object in the 3D model.

[0087] In certain embodiments, a simple solution may be to use maximum likelihood (ML) estimation by assigning to each segment a label that dominates the corresponding segment. ML estimation may not produce an optimal solution. The correct label may be rejected due to a slightly lower frequency of occurrence. To address this issue, an estimate of label confidence based on semantic meaning and category world assumptions may be used. As an example and not limitation, if the 3D scene represents an apartment in Manhattan, the category world may be the Manhattan world, which includes information corresponding to objects typically found in Manhattan apartments. In certain embodiments, an object may be associated with multiple environments. As an example and not limitation, using Mask-RCNN segmentation, the gravity direction may be first estimated by taking the dominant segment as the marker floor. A least squares plane fit may then be performed to obtain the parameters (ng, dg). The estimated plane normal may be used as the gravity direction. For the primitive p predicted to be the floor i , the label confidence can be estimated by

[0088]

[0089] As an example and not limitation, assuming that the reconstruction can be a Manhattan world, objects that are generally perpendicular to the direction of gravity (such as tables and ceilings) can be assigned confidences However, objects that are generally parallel to gravity (such as walls, doors, and monitors) can be assigned confidence For object classes whose surface orientation may be difficult to describe by the Manhattan world, the confidence can be set to an empirical value that reflects the overall prediction confidence. Using the confidence value, weighted voting can be performed in the label transfer, which can lead to improved labeling. In a specific embodiment, the system can calculate the confidence value of the information projected onto the 3D model. As an example and not limitation, the system can calculate the confidence value that the semantic information of the object from the semantic segmentation corresponds to the corresponding object. That is, the confidence value of whether the detected object is indeed a chair. In a specific embodiment, the confidence value can be based on whether the semantic information of the object corresponds to an object within a specific environment. As an example and not limitation, in the Manhattan world category, typical objects can be found in the environment. For example, coffee tables may be common in the Manhattan world category compared to other world categories. In addition, certain configurations or styles of specific objects can be found in the environment. As an example and not limitation, modern chairs can be found in the Manhattan world category, but other types of chairs may be uncommon. In a specific embodiment, the system can project information onto the 3D model based on whether the determined confidence value exceeds a threshold confidence value. In certain embodiments, if the determined confidence value of the information is less than a threshold confidence value, the system may discard the information.In certain embodiments, the projection of the information to the 3D model may include marking corresponding objects in the 3D model.

[0090] In certain embodiments, the system may use annotated 3D models to accurately render 3D scenes. By way of example and not limitation, the system may correctly identify a rug as separate from a floor and render them separately from each other. In certain embodiments, the system may accurately render augmented reality elements over a real world view by using semantic information about objects. By way of example and not limitation, the system may generate an augmented reality table lamp that is above a unique table identified within a real world view. In certain embodiments, the system may use the semantic meaning of objects to filter objects from a virtual reality environment. By way of example and not limitation, a user may remove an object from a semantic set "furniture" in a virtual reality environment.

[0091] Figure 7 An example network environment 700 associated with a virtual reality system is shown. The network environment 700 includes a user 701 interacting with a client system 730, a social networking system 760, and a third-party system 770, which are connected to each other via a network 710. Figure 7A particular arrangement of user 701, client system 730, social networking system 760, third-party system 770, and network 710 is shown, but the present disclosure contemplates any suitable arrangement of user 701, client system 730, social networking system 760, third-party system 770, and network 710. By way of example and not limitation, two or more of user 701, client system 730, social networking system 760, and third-party system 770 may bypass network 710 and connect directly to each other. As another example, two or more of client system 730, social networking system 760, and third-party system 770 may be physically or logically co-located with each other in whole or in part. Furthermore, although Figure 7 A particular number of users 701, client systems 730, social networking systems 760, third-party systems 770, and networks 710 are shown, but the present disclosure contemplates any suitable number of client systems 730, social networking systems 760, third-party systems 770, and networks 710. By way of example and not limitation, network environment 700 may include a plurality of users 701, client systems 730, social networking systems 760, third-party systems 770, and networks 710.

[0092] The present disclosure contemplates any suitable network 710. By way of example and not limitation, one or more portions of network 710 may include an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of a public switched telephone network (PSTN), a cellular telephone network, or a combination of two or more of these. Network 710 may include one or more networks 710.

[0093] Links 750 may connect client systems 730, social networking systems 760, and third-party systems 770 to communication network 710 or to each other. The present disclosure contemplates any suitable links 750. In certain embodiments, one or more links 750 include one or more wired (such as, for example, digital subscriber lines (DSL) or cable-based data service interface specifications (DOCSIS)) links, wireless (such as, for example, Wi-Fi or Worldwide Interoperability for Microwave Access (WiMAX)) links, or optical (such as, for example, synchronous optical network (SONET) or synchronous digital hierarchy (SDH)) links. In certain embodiments, one or more links 750 each include a self-organizing network, an intranet, an extranet, a VPN, a LAN, a WLAN, a WAN, a WWAN, a MAN, a portion of the Internet, a portion of the PSTN, a network based on cellular technology, a network based on satellite communication technology, another link 750, or a combination of two or more such links 750. Links 750 need not necessarily be the same throughout the network environment 700. One or more first links 750 may be different from one or more second links 750 in one or more aspects.

[0094] In certain embodiments, client system 730 may be an electronic device that includes hardware, software, or embedded logic components, or a combination of two or more such components, and is capable of performing appropriate functions implemented or supported by client system 730. By way of example and not limitation, client system 730 may include a computer system such as a desktop computer, a notebook or laptop computer, a netbook, a tablet computer, an e-book reader, a GPS device, a camera, a personal digital assistant (PDA), a handheld electronic device, a cellular phone, a smart phone, a virtual reality head-mounted device and controller, other suitable electronic devices, or any suitable combination thereof. The present disclosure contemplates any suitable client system 730. Client system 730 may enable network users at client system 730 to access network 710. Client system 730 may enable its users to communicate with other users at other client systems 730. Client system 730 may generate a virtual reality environment for users to interact with content.

[0095] In certain embodiments, the client system 730 may include a virtual reality (or augmented reality) head mounted device 732 (such as OCULUS RIFT, etc.) and a virtual reality input device 734 (such as a virtual reality controller). A user at the client system 730 may wear the virtual reality head mounted device 732 and use the virtual reality input device to interact with the virtual reality environment 736 generated by the virtual reality head mounted device 732. Although not shown, the client system 730 may also include a separate processing computer and / or any other component of the virtual reality system. The virtual reality head mounted device 732 may generate a virtual reality environment 736, which may include system content 738 (including but not limited to an operating system) (such as software or firmware updates), and also include third-party content 740, such as content from an application or content dynamically downloaded from the Internet (e.g., web content). The virtual reality head mounted device 732 may include a sensor 742 (such as an accelerometer, a gyroscope, a magnetometer) to generate sensor data that tracks the positioning of the head mounted device 732. The head mounted device 732 may also include an eye tracker for tracking the position of the user's eyes or their viewing direction. The client system may use data from the sensors 742 to determine velocity, orientation, and gravity about the head mounted device. The virtual reality input device 734 may include sensors 744 (such as accelerometers, gyroscopes, magnetometers, and touch sensors) to generate sensor data that tracks the location of the input device 734 and the position of the user's fingers. The client system 730 may utilize outside-in tracking, where a tracking camera (not shown) is placed outside of the virtual reality head mounted device 732 and within the line of sight of the virtual reality head mounted device 732. In outside-in tracking, the tracking camera may track the location of the virtual reality head mounted device 732 (e.g., by tracking one or more infrared LED markers on the virtual reality head mounted device 732). Alternatively or additionally, the client system 730 may utilize inside-out tracking, where a tracking camera (not shown) may be placed on or within the virtual reality head mounted device 732 itself. In inside-out tracking, a tracking camera captures images of its surroundings in the real world, and can use the changing perspective of the real world to determine its own position in space.

[0096] The third-party content 740 may include a web browser (such as MICROSOFT INTERNET EXPLORER, GOOGLECHROME or MOZILLA FIREFOX), and may have one or more add-ons, plug-ins or other extensions (such as TOOLBAR or YAHOO TOOLBAR). A user at the client system 730 may enter a uniform resource locator (URL) or other address that directs the web browser to a specific server (such as server 762 or a server associated with the third-party system 770), and the web browser may generate a hypertext transfer protocol (HTTP) request and pass the HTTP request to the server. The server may accept the HTTP request and pass one or more hypertext markup language (HTML) files to the client system 730 in response to the HTTP request. The client system 730 may display a web interface (such as a web page) based on the HTML file from the server for presentation to the user. The present disclosure contemplates any suitable source file. As an example and not as a limitation, a web interface may be displayed according to an HTML file, an extensible hypertext markup language (XHTML) file or an extensible markup language (XML) file according to specific needs. Such an interface may also execute scripts, such as, for example and without limitation, scripts written in JAVASCRIPT, JAVA, MICROSOFT SILVERLIGHT, a combination of markup language and script (e.g., AJAX (Asynchronous JAVASCRIPT and XML)), etc. Herein, references to a web interface include one or more corresponding source files (which a browser may use to render the web interface), and vice versa, where appropriate.

[0097] In certain embodiments, social networking system 760 may be a network addressable computing system that may host an online social network. Social networking system 760 may generate, store, receive, and send social networking data (such as, for example, user profile data, concept profile data, social graph information, or other suitable data related to an online social network). Social networking system 760 may be accessed by other components of network environment 700 directly or via network 710. As an example and not by way of limitation, client system 730 may access social networking system 760 directly or via network 710 using a web browser of third-party content 740 or a native application associated with social networking system 760 (e.g., a mobile social networking application, a messaging application, another suitable application, or any combination thereof). In certain embodiments, social networking system 760 may include one or more servers 762. Each server 762 may be a unitary server or a distributed server across multiple computers or multiple data centers. Server 762 can have various types, for example and without limitation, web server, news server, mail server, message server, advertising server, file server, application server, exchange server, database server, proxy server, another server suitable for performing the functions or processes described herein, or any combination thereof. In a particular embodiment, each server 762 may include hardware, software or embedded logic components, or a combination of two or more such components for performing the appropriate functions implemented or supported by server 762. In a particular embodiment, social networking system 760 may include one or more data storage 764. Data storage 764 may be used to store various types of information. In a particular embodiment, the information stored in data storage 764 may be organized according to a particular data structure. In a particular embodiment, each data storage 764 may be a relational database, a columnar database, a correlation database, or other suitable database. Although the present disclosure describes or illustrates a particular type of database, the present disclosure contemplates any suitable type of database. Particular embodiments may provide an interface that enables client system 730 , social networking system 760 , or third-party system 770 to manage, retrieve, modify, add, or delete information stored in data store 764 .

[0098] In certain embodiments, social networking system 760 may store one or more social graphs in one or more data stores 764. In certain embodiments, a social graph may include a plurality of nodes, which may include a plurality of user nodes (each corresponding to a particular user) or a plurality of concept nodes (each corresponding to a particular concept) and a plurality of edges connecting the nodes. Social networking system 760 may provide users of an online social network with the ability to communicate and interact with other users. In certain embodiments, a user may join an online social network via social networking system 760 and then add connections (e.g., relationships) with a plurality of other users in social networking system 760 to whom they want to be connected. Herein, the term "friend" may refer to any other user of social networking system 760 with whom the user forms a connection, association, or relationship via social networking system 760.

[0099] In particular embodiments, social-networking system 760 may provide users with the ability to take actions on various types of items or objects supported by social-networking system 760. By way of example and not limitation, items and objects may include groups or social networks to which users of social-networking system 760 may belong, events or calendar entries that may be of interest to users, computer-based applications that users may use, transactions that allow users to buy or sell goods via a service, interactions with advertisements that users may perform, or other suitable items or objects. Users may interact with anything that can be represented in social-networking system 760 or by an external system such as third-party system 770, which is separate from social-networking system 760 and coupled to social-networking system 760 via network 710.

[0100] In certain embodiments, social networking system 760 can link various entities. As an example and not by way of limitation, social networking system 760 can enable users to interact with each other and receive content from third-party systems 770 or other entities, or allow users to interact with these entities through an application programming interface (API) or other communication channels.

[0101] In certain embodiments, third-party system 770 may include one or more types of servers, one or more data stores, one or more interfaces (including but not limited to APIs), one or more web services, one or more content sources, one or more networks, or any other suitable components (e.g., the server may communicate with these components). Third-party system 770 may be operated by a different entity than the entity operating social networking system 760. However, in certain embodiments, social networking system 760 and third-party system 770 may operate in conjunction with each other to provide social networking services to users of social networking system 760 or third-party system 770. In this sense, social networking system 760 may provide a platform or backbone that other systems (e.g., third-party system 770) may use to provide social networking services and functionality to users across the Internet.

[0102] In certain embodiments, third-party system 770 may include a third-party content object provider. A third-party content object provider may include one or more sources of content objects that may be delivered to client system 730. For example, and not by way of limitation, a content object may include information about things or activities of interest to a user, such as movie showtimes, movie reviews, restaurant reviews, restaurant menus, product information and reviews, or other suitable information. As another example and not by way of limitation, a content object may include an incentive content object (e.g., a coupon, discount voucher, gift certificate, or other suitable incentive object).

[0103] In certain embodiments, social networking system 760 also includes user-generated content objects that can enhance a user's interaction with social networking system 760. User-generated content can include any content that a user can add, upload, send, or "post" to social networking system 760. As an example and not by way of limitation, a user transmits a post from client system 730 to social networking system 760. Posts can include data, such as status updates or other textual data, location information, photos, videos, links, music, or other similar data or media. Content can also be added to social networking system 760 by third parties through "communication channels" such as dynamic messages or streams.

[0104] In a particular embodiment, the social networking system 760 may include various servers, subsystems, programs, modules, logs, and data storage. In a particular embodiment, the social networking system 760 may include one or more of the following: a web server, an action recorder, an API request server, a correlation and ranking engine, a content object classifier, a notification controller, an action log, a third-party content object exposure log, an inference module, an authorization / privacy server, a search module, an advertisement-targeting module, a user interface module, a user profile storage, a connection storage, a third-party content storage, or a positioning storage. The social networking system 760 may also include suitable components, such as a network interface, a security mechanism, a load balancer, a failover server, a management and network operation console, other suitable components, or any suitable combination thereof. In a particular embodiment, the social networking system 760 may include one or more user profile storages for storing user profiles. A user profile may include, for example, biographical information, demographic information, behavioral information, social information, or other types of descriptive information (e.g., work experience, educational history, hobbies or preferences, interests, affinities, or positioning). Interest information may include interests associated with one or more categories. Categories may be general or specific. As an example and not by way of limitation, if a user "likes" an article about a brand of shoes, the category may be the brand, or the general category of "shoes" or "clothing". The association storage may be used to store association information about users. The association information may indicate users with similar or common work experience, group membership, hobbies, educational history, or who are related or share common attributes in any way. The association information may also include user-defined associations between different users and content (internal and external). The web server may be used to link the social networking system 760 to one or more client systems 730 or one or more third-party systems 770 via the network 710. The web server may include a mail server or other messaging functionality for receiving and routing messages between the social networking system 760 and one or more client systems 730. The API request server may allow a third-party system 770 to access information from the social networking system 760 by calling one or more APIs. The action recorder may be used to receive communications from the web server about the user's actions on or outside the social networking system 760. In conjunction with the action log, a third-party content object log of user exposure to third-party content objects can be maintained. The notification controller can provide information about content objects to the client system 730. The information can be pushed to the client system 730 as a notification, or the information can be pulled from the client system 730 in response to a request received from the client system 730.The authorization server may be used to enforce one or more privacy settings of users of the social networking system 760. The privacy settings of the user determine how certain information associated with the user may be shared. The authorization server may allow the user to opt-in or opt-out of having their actions recorded by the social networking system 760 or shared with other systems (e.g., third-party systems 770), such as by setting appropriate privacy settings. A third-party content object store may be used to store content objects received from a third party (e.g., third-party system 770). A location store may be used to store location information received from a client system 730 associated with the user. The advertising pricing module may combine social information, current time, location information, or other suitable information to provide relevant advertisements to the user in the form of notifications.

[0105] Figure 8 An example method 800 for generating a semantic model is shown. In certain embodiments, a virtual reality head mounted device 732 may be used to generate a semantic model. The method may begin at step 810, where a computing system (e.g., a virtual reality system) may access a plurality of images captured by one or more cameras from a plurality of camera poses. At step 820, the computing system may use the plurality of images to generate a plurality of semantic segmentations including semantic information of one or more objects captured in the plurality of images. At step 830, the computing system may access a three-dimensional (3D) model of the one or more objects. At step 840, the computing system may use the plurality of camera poses to determine a corresponding plurality of virtual camera poses relative to the 3D model of the one or more objects. At step 850, the computing system may generate a semantic 3D model by projecting the semantic information of the plurality of semantic segmentations onto the 3D model using the plurality of virtual camera poses. Where appropriate, certain embodiments may be repeated. Figure 8 Although the present disclosure describes and illustrates one or more steps of the method. Figure 5 The specific steps of the method occur in a specific order, but the present disclosure contemplates Figure 8 Any suitable steps of the method may appear in any suitable order. In addition, although the present disclosure describes and illustrates methods including Figure 8 The present disclosure contemplates any suitable method for generating a semantic model including any suitable steps, and where appropriate, the steps may include: Figure 8 All, some, or none of the steps of the method Figure 8 Furthermore, although the present disclosure describes and illustrates the steps of performing Figure 8 specific components, devices or systems for specific steps of the method, but the present disclosure contemplates performing Figure 8 Any suitable combination of any suitable components, devices, or systems of any suitable steps of the method.

[0106] Although the present disclosure describes and illustrates the process of performing various functions in the environment of the virtual reality head mounted device 132, a computing system (e.g., a server embodied as a social networking system 160 or a third-party system 170) may handle the processing and send the results to the virtual reality head mounted device 132, and vice versa.

[0107] Fig. 9 An example computer system 900 is shown. In a particular embodiment, one or more computer systems 900 perform one or more steps of one or more methods described or shown herein. In a particular embodiment, one or more computer systems 900 provide the functionality described or shown herein. In a particular embodiment, software running on one or more computer systems 900 performs one or more steps of one or more methods described or shown herein, or provides the functionality described or shown herein. A particular embodiment includes one or more parts of one or more computer systems 900. Herein, references to computer systems may include computing devices, and vice versa, where appropriate. In addition, references to computer systems may include one or more computer systems, where appropriate.

[0108] The present disclosure contemplates any suitable number of computer systems 900. The present disclosure contemplates that the computer system 900 takes any suitable physical form. As an example and not by way of limitation, the computer system 900 may be an embedded computer system, a system on a chip (SOC), a single board computer system (SBC) (such as, for example, a computer on a module (COM) or a system on a module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a mesh of computer systems, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, or a combination of two or more of these. Where appropriate, the computer system 900 may include one or more computer systems 900; be monolithic or distributed; span multiple locations; span multiple machines; span multiple data centers; or reside in a cloud, which may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems 900 may perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitations. As an example and not as a limitation, one or more computer systems 900 may perform one or more steps of one or more methods described or shown herein in real time or in batch mode. Where appropriate, one or more computer systems 900 may perform one or more steps of one or more methods described or shown herein at different times or at different locations.

[0109] In a particular embodiment, computer system 900 includes a processor 902, a memory 904, a storage device 906, an input / output (I / O) interface 908, a communication interface 910, and a bus 912. Although this disclosure describes and illustrates a particular computer system having particular numbers of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable numbers of any suitable components in any suitable arrangement.

[0110] In a particular embodiment, the processor 902 includes hardware for executing instructions (e.g., those that constitute a computer program). As an example and not by way of limitation, to execute instructions, the processor 902 may retrieve (or fetch) instructions from an internal register, an internal cache, a memory 904, or a storage device 906; decode them and execute them; and then write one or more results to an internal register, an internal cache, a memory 904, or a storage device 906. In a particular embodiment, the processor 902 may include one or more internal caches for data, instructions, or addresses. Where appropriate, the present disclosure contemplates that the processor 902 includes any suitable number of any suitable internal caches. As an example and not by way of limitation, the processor 902 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). The instructions in the instruction cache may be copies of the instructions in the memory 904 or the storage device 906, and the instruction cache may speed up the retrieval of those instructions by the processor 902. The data in the data cache may be: a copy of data in the memory 904 or storage device 906 for operating the instructions executed at the processor 902; the result of the previous instruction executed at the processor 902, for access by the subsequent instruction executed at the processor 902 or for writing to the memory 904 or storage device 906; or other suitable data. The data cache may speed up the read or write operation performed by the processor 902. The TLB may speed up the virtual address translation for the processor 902. In a particular embodiment, the processor 902 may include one or more internal registers for data, instructions, or addresses. Where appropriate, the present disclosure contemplates that the processor 902 includes any suitable number of any suitable internal registers. Where appropriate, the processor 902 may include one or more arithmetic logic units (ALUs); be a multi-core processor; or include one or more processors 902. Although the present disclosure describes and illustrates a particular processor, the present disclosure contemplates any suitable processor.

[0111] In a particular embodiment, the memory 904 includes a main memory for storing instructions for the processor 902 to execute or data for the processor 902 to operate. As an example and not by way of limitation, the computer system 900 may load instructions from the storage device 906 or another source (e.g., another computer system 900) to the memory 904. The processor 902 may then load the instructions from the memory 904 to an internal register or an internal cache. In order to execute the instructions, the processor 902 may retrieve the instructions from the internal register or the internal cache and decode them. During or after the execution of the instructions, the processor 902 may write one or more results (which may be intermediate results or final results) to the internal register or the internal cache. The processor 902 may then write one or more of these results to the memory 904. In a particular embodiment, the processor 902 only executes instructions in one or more internal registers or internal caches or in the memory 904 (rather than the storage device 906 or elsewhere), and only operates on data in one or more internal registers or internal caches or in the memory 904 (rather than the storage device 906 or elsewhere). One or more memory buses (which may each include an address bus and a data bus) may couple the processor 902 to the memory 904. As described below, the bus 912 may include one or more memory buses. In a particular embodiment, one or more memory management units (MMUs) reside between the processor 902 and the memory 904 and facilitate access to the memory 904 requested by the processor 902. In a particular embodiment, the memory 904 includes a random access memory (RAM). Where appropriate, the RAM may be a volatile memory. Where appropriate, the RAM may be a dynamic RAM (DRAM) or a static RAM (SRAM). In addition, where appropriate, the RAM may be a single-port RAM or a multi-port RAM. The present disclosure contemplates any suitable RAM. Where appropriate, the memory 904 may include one or more memories 904. Although the present disclosure describes and illustrates specific memories, the present disclosure contemplates any suitable memory.

[0112] In a particular embodiment, the storage device 906 includes a large-capacity storage device for data or instructions. As an example and not as a limitation, the storage device 906 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the storage device 906 may include a removable or non-removable (or fixed) medium. Where appropriate, the storage device 906 may be inside or outside the computer system 900. In a particular embodiment, the storage device 906 is a non-volatile solid-state memory. In a particular embodiment, the storage device 906 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically variable ROM (EAROM), or a flash memory, or a combination of two or more of these. The present disclosure contemplates a large-capacity storage device 906 in any suitable physical form. Where appropriate, the storage device 906 may include one or more storage device control units that facilitate communication between the processor 902 and the storage device 906. Where appropriate, storage 906 may include one or more storage devices 906. Although this disclosure describes and illustrates particular storage devices, this disclosure contemplates any suitable storage devices.

[0113] In a particular embodiment, the I / O interface 908 includes hardware, software, or both that provide one or more interfaces for communication between the computer system 900 and one or more I / O devices. Where appropriate, the computer system 900 may include one or more of these I / O devices. One or more of these I / O devices can implement communication between a person and the computer system 900. As an example and not as a limitation, the I / O device may include a keyboard, a keypad, a microphone, a monitor, a mouse, a printer, a scanner, a speaker, a static camera, a stylus, a tablet computer, a touch screen, a tracking ball, a video camera, another suitable I / O device, or a combination of two or more of these. The I / O device may include one or more sensors. The present disclosure contemplates any suitable I / O device and any suitable I / O interface 908 for them. Where appropriate, the I / O interface 908 may include one or more devices or software drivers that enable the processor 902 to drive one or more of these I / O devices. Where appropriate, the I / O interface 908 may include one or more I / O interfaces 908. Although this disclosure describes and illustrates a particular I / O interface, this disclosure contemplates any suitable I / O interface.

[0114] In a particular embodiment, the communication interface 910 includes hardware, software, or both that provide one or more interfaces for communication (e.g., packet-based communication) between the computer system 900 and one or more other computer systems 900 or one or more networks. As an example and not as a limitation, the communication interface 910 may include a network interface controller (NIC) or a network adapter for communicating with an Ethernet or other wired network, or a wireless NIC (WNIC) or a wireless adapter for communicating with a wireless network (e.g., a Wi-Fi network). The present disclosure contemplates any suitable network and any suitable communication interface 910 for it. As an example and not as a limitation, the computer system 900 may communicate with one or more parts of an ad hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or the Internet, or a combination of two or more of these. One or more parts of one or more of these networks may be wired or wireless. As an example, the computer system 900 may communicate with a wireless PAN (WPAN) (e.g., a Bluetooth WPAN), a WI-FI network, a WI-MAX network, a cellular telephone network (e.g., a Global System for Mobile Communications (GSM) network), or other suitable wireless networks, or a combination of two or more of these. Where appropriate, the computer system 900 may include any suitable communication interface 910 for any of these networks. Where appropriate, the communication interface 910 may include one or more communication interfaces 910. Although this disclosure describes and illustrates particular communication interfaces, this disclosure contemplates any suitable communication interface.

[0115] In a particular embodiment, bus 912 includes hardware, software, or both that couples components of computer system 900 to each other. As an example and not by way of limitation, bus 912 may include an accelerated graphics port (AGP) or other graphics bus, an extended industry standard architecture (EISA) bus, a front side bus (FSB), a HYPERTRANSPORT (HT) interconnect, an industry standard architecture (ISA) bus, an INFINIBAND interconnect, a low pin count (LPC) bus, a memory bus, a micro channel architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI-Express (PCIe) bus, a serial advanced technology attachment (SATA) bus, a video electronics standards association local (VLB) bus, or any other suitable bus, or a combination of two or more of these. Where appropriate, bus 912 may include one or more buses 912. Although the present disclosure describes and illustrates a particular bus, the present disclosure contemplates any suitable bus or interconnect.

[0116] Herein, where appropriate, one or more computer-readable non-transitory storage media may include one or more semiconductor-based or other integrated circuits (ICs) (e.g., field programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard disk drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical disk drives, floppy disks, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM drives, secure digital (SECURE DIGITAL) cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these. Where appropriate, the computer-readable non-transitory storage media may be volatile, non-volatile, or a combination of volatile and non-volatile.

[0117] As used herein, "or" is inclusive and not exclusive, unless expressly indicated otherwise or indicated by context. Thus, as used herein, "A or B" means "A, B, or both," unless expressly indicated otherwise or indicated by context. Furthermore, "and" is both conjunctive and several, unless expressly indicated otherwise or indicated by context. Thus, as used herein, "A and B" means "A and B, jointly or severally," unless expressly indicated otherwise or indicated by context.

[0118] The scope of the present disclosure includes all changes, substitutions, variations, alterations and modifications to the example embodiments described or shown herein that will be understood by those of ordinary skill in the art. The scope of the present disclosure is not limited to the example embodiments described or shown herein. In addition, although the present disclosure describes and illustrates the corresponding embodiments herein as including specific components, elements, features, functions, operations or steps, any of these embodiments may include any combination or replacement of any components, elements, features, functions, operations or steps described or shown anywhere herein that will be understood by those of ordinary skill in the art. In addition, references to devices or systems or components of devices or systems that are suitable for, arranged to, capable of, configured to, implemented to, operable to, or operated to perform specific functions in the appended claims include the device, system, component, whether it or that specific function is activated, turned on or unlocked, as long as the device, system or component is adjusted, arranged, enabled, configured, implemented, operable, or operated in this way. In addition, although the present disclosure describes or illustrates specific embodiments as providing specific advantages, specific embodiments may provide some, all or none of these advantages.

Claims

1. A method for rendering a virtual or augmented reality environment and interacting with the virtual or augmented reality environment, comprising: accessing a plurality of images captured by one or more cameras from a plurality of camera poses; generating, using the plurality of images, a plurality of semantic segmentations comprising semantic information of one or more objects captured in the plurality of images; accessing a three-dimensional (3D) model of the one or more objects; determining a corresponding plurality of virtual camera poses relative to the 3D model of the one or more objects using the plurality of camera poses; A semantic 3D model is generated by projecting semantic information of the multiple semantic segmentations to the 3D model using the multiple virtual camera poses, wherein semantic information from two or more of the multiple semantic segmentations is combined to be applied to a first object among the one or more objects, and wherein each of the multiple semantic segmentations corresponds to each of the multiple virtual camera poses. 2 . The method of claim 1 , further comprising generating, using the plurality of images, a plurality of geometry-based segmentations comprising geometric information of the one or more objects captured in the plurality of images. 3 . The method of claim 1 , further comprising generating a plurality of instance segmentations using the plurality of images, the plurality of instance segmentations comprising object identifications of the one or more objects captured in the plurality of images. 4 . The method of claim 3 , further comprising generating an instance 3D model by projecting the object identities of the plurality of instance segmentations onto the 3D model using the plurality of virtual camera poses.

5. The method according to claim 1, further comprising generating a semantic tree based on semantic information of the one or more objects captured in the plurality of images, wherein The semantic tree includes a plurality of semantic levels, and wherein each semantic level corresponds to a threshold amount of semantic information.

6. The method according to claim 5, wherein: The multiple semantic levels include one or more of the following: a first semantic level, the first semantic level comprising information corresponding to each of the one or more objects, a second semantic level, the second semantic level comprising information of object classes corresponding to the one or more objects, and A third semantic level includes information corresponding to a semantic set of the one or more objects.

7. The method according to claim 5, wherein: Projecting the semantic information further includes projecting semantic information of a specific semantic level onto the 3D model.

8. The method according to claim 1, wherein: A 3D model of the one or more objects is generated based on the plurality of images. 9 . The method according to claim 1 , further comprising calculating one or more confidence values ​​that each semantic information of the one or more objects from the multiple semantic segmentations corresponds to a corresponding object.

10. The method according to claim 9, wherein: The objects are associated with a plurality of environments, and wherein the one or more confidence values ​​are determined based on whether the semantic information of the one or more objects corresponds to the object within a particular environment.

11. The method according to claim 9, wherein: Projecting the semantic information of the plurality of semantic segments onto the 3D model further comprises: determining that a confidence value of semantic information of an object exceeds a threshold confidence value; and In response to the confidence value exceeding the threshold confidence value, the semantic information is projected to a 3D model of the corresponding object.

12. The method according to claim 9, wherein: Semantic information associated with confidence values ​​less than a threshold confidence value is discarded.

13. The method according to claim 1, wherein: Generating the semantic 3D model includes combining semantic information from each of the plurality of semantic segmentations to apply to one of the one or more objects, and / or wherein projecting the semantic information to the 3D model includes adding a label corresponding to a respective object in the 3D model.

14. The method according to claim 2, wherein: Generating the semantic 3D model by projecting the semantic information of the plurality of semantic segmentations further includes projecting semantic information corresponding to the one or more objects captured in the plurality of images using geometric information of the one or more objects.

15. The method according to claim 4, wherein: Generating the instance 3D model includes combining object identification from each of the plurality of instance segmentations to apply to one of the one or more objects.

16. One or more computer-readable non-transitory storage media containing software that, when executed, is operable to: accessing a plurality of images captured by one or more cameras from a plurality of camera poses; generating, using the plurality of images, a plurality of semantic segmentations comprising semantic information of one or more objects captured in the plurality of images; accessing a three-dimensional (3D) model of the one or more objects; determining a corresponding plurality of virtual camera poses relative to the 3D model of the one or more objects using the plurality of camera poses; A semantic 3D model is generated by projecting semantic information of the multiple semantic segmentations to the 3D model using the multiple virtual camera poses, wherein semantic information from two or more of the multiple semantic segmentations is combined to be applied to a first object among the one or more objects, and wherein each of the multiple semantic segmentations corresponds to each of the multiple virtual camera poses.

17. A system for rendering a virtual or augmented reality environment and interacting with the virtual or augmented reality environment, comprising: one or more processors; and a non-transitory memory coupled to the processor, the non-transitory memory comprising instructions executable by the processor, the processor, when executing the instructions, being operable to: accessing a plurality of images captured by one or more cameras from a plurality of camera poses; generating, using the plurality of images, a plurality of semantic segmentations comprising semantic information of one or more objects captured in the plurality of images; accessing a three-dimensional (3D) model of the one or more objects; determining a corresponding plurality of virtual camera poses relative to the 3D model of the one or more objects using the plurality of camera poses; A semantic 3D model is generated by projecting semantic information of the multiple semantic segmentations to the 3D model using the multiple virtual camera poses, wherein semantic information from two or more of the multiple semantic segmentations is combined to be applied to a first object among the one or more objects, and wherein each of the multiple semantic segmentations corresponds to each of the multiple virtual camera poses.

Images