World Space Split

Abstract

Methods, systems, and apparatus for generating a persistent world-space ground (or floor) segmentation map (or "texture") for use in augmented or virtual reality 3D experiences.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of priority to U.S. patent application serial number 16 / 688,708, filed on November 19, 2019, entitled “WORLD-SPACE SEGMENTATION,” the entire contents of which are incorporated herein by reference. Background Art

[0003] Recent advances in computing have enabled consumer electronic devices, such as smartphones and tablets, to generate and display increasingly detailed mixed reality environments. The usefulness of many mixed reality effects depends on correctly identifying the categories of objects depicted in image frames captured by the device. To provide a stable foundation for certain mixed reality effects, it is desirable to identify portions of each frame as, for example, "floor," "ground," or "sky." BRIEF DESCRIPTION OF THE DRAWINGS

[0004] To easily identify the discussion of any particular element or act, the highest digit or digits in a reference number refer to the figure number in which the element is first introduced.

[0005] Figure 1 is a pictorial representation of a networked environment in which the present disclosure may be deployed, according to some example implementations.

[0006] Figure 2 is a graphical representation of a data structure maintained in a database according to some example embodiments.

[0007] Figure 3 is a pictorial representation of a processing environment according to some example embodiments.

[0008] Figure 4 is a block diagram illustrating a software architecture in which the present disclosure may be implemented, according to some example embodiments.

[0009] Figure 5 is a diagrammatic representation of a machine in the form of a computer system within which a set of instructions may be executed, for causing the machine to perform any one or more of the methodologies discussed, according to some example embodiments.

[0010] Figure 6 Main operational blocks of a routine for performing semantic segmentation of an image frame are shown, according to some example embodiments.

[0011] Figure 7 Main operational blocks in carrying out the task of generating user interface data for display are shown according to some example embodiments.

[0012] Figure 8Main operational blocks in implementing the task of determining segmentation are shown according to some example embodiments.

[0013] Figure 9 A “ground truth” street scene 900 is shown according to an exemplary embodiment.

[0014] Figure 10 According to one embodiment, Figure 9 A street scene application where the road surface is textured to resemble lava 1000 for a virtually modified ground effect.

[0015] Figure 11 According to another embodiment, Figure 9 The street scene applies a ground effect in which the height of the floor is dynamically changed to depict a virtual modification of the undulating water surface 1100 . DETAILED DESCRIPTION

[0016] In many cases, it is convenient to attach object classifications to blocks of pixels (or even to individual pixels) within each image frame; this process is known as segmentation of the image frame.

[0017] Many techniques are available for image segmentation. One increasingly popular technique is semantic segmentation, which uses a neural network to determine the likely association between a particular patch of pixels in an image frame and a label, such as "ground." The neural network is trained to identify the likely semantic classifications of pixels in a captured image frame (i.e., the labels a human observer might assign to those pixels). The image is thus segmented into pixels that belong to a particular classification and pixels that do not.

[0018] Semantic segmentation neural networks typically operate in screen space (i.e., the pixels to which the segmentation process is applied are the pixels rendered on the device's display). Generating mixed reality video effects based on the output of conventional semantic segmentation processing is difficult because these effects are typically sensitive to relatively small temporal instabilities in the image frames captured for display.

[0019] Conventional semantic segmentation neural networks are necessarily limited to the camera angle at which the image frames were captured: move the camera angle or reposition the camera relative to the depicted scene, and the network must be reapplied to achieve a useful segmentation.

[0020] Conventional segmentation computations introduce a significant burden in terms of computational power, which sets a high computational power requirement threshold, effectively preventing many devices (typically low-end or legacy devices) from performing the desired effect.

[0021] Implementations of the present disclosure address these and other problems.

[0022] The following description includes the system, method, technology, instruction sequence and computing machine program product of the illustrative embodiment of the present disclosure. In the following description, for the purpose of illustration, many specific details are set forth to provide an understanding of the various embodiments of the present invention. However, it will be apparent to those skilled in the art that the embodiments of the present invention can be put into practice without these specific details. Usually, it is not necessary to show in detail known instruction examples, protocols, structures and technologies.

[0023] Embodiments of the present disclosure provide methods for generating a persistent world-space ground (or floor) segmentation map (or "texture") for use in augmented or virtual reality 3D experiences.

[0024] Known semantic segmentation neural networks operate in screen space and are therefore difficult to use in real-time with virtual 3D objects due to temporal instability and the inflexibility of sampling images only in screen space. To address temporal instability, it is necessary to run a semantic segmentation neural network on each consecutive frame.

[0025] In some embodiments, mixed reality effects are applied to captured video frames in real time. The practicality of such effects may be severely compromised by the need to provide accurate segmentation of the consecutive video frames on which the effect is based. Many capture devices may not be able to handle conventional segmentation at typical frame rates.

[0026] In some embodiments, features in the captured video frame are completely replaced by virtual data in the display, however it may be convenient to preserve aspects of the real world space, for example, the level of the world space floor relative to the viewpoint of the image capture device can be preserved as the level of the virtual floor relative to the viewpoint of the virtual reality display.

[0027] Once the image frame data is captured and the mixed reality effects are applied, the resulting multimedia stream can be stored for later transmission to other devices, such as via a messaging system. Alternatively or additionally, the multimedia stream can be transmitted in real time according to a real-time communication protocol, so that more than one communicatively connected device can participate in a shared mixed reality experience.

[0028] Figure 1 1 is a block diagram illustrating an example messaging system 100 for exchanging data (e.g., messages and associated content) over a network. The messaging system 100 includes multiple instances of client devices 102, each client device 102 hosting multiple applications including a messaging client application 104. Each messaging client application 104 is communicatively coupled to other instances of the messaging client application 104 and a messaging server system 108 via a network 106 (e.g., the Internet).

[0029] The messaging client application 104 is able to communicate and exchange data with another messaging client application 104 and with a messaging server system 108 via the network 106. The data exchanged between the messaging client applications 104 and between the messaging client applications 104 and the messaging server system 108 includes functions (e.g., commands to invoke functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0030] The messaging server system 108 provides server-side functionality to particular messaging client applications 104 via the network 106. Although certain functionality of the messaging system 100 is described herein as being performed by either the messaging client application 104 or the messaging server system 108, the location of certain functionality within the messaging client application 104 or within the messaging server system 108 is a design choice. For example, it may be technically preferable to initially deploy certain technologies and functionality within the messaging server system 108 but later migrate the technologies and functionality to the messaging client application 104 if the client device 102 has sufficient processing power.

[0031] The messaging server system 108 supports various services and operations provided to the messaging client applications 104. Such operations include sending data to the messaging client applications 104, receiving data from the messaging client applications 104, and processing data generated by the messaging client applications 104. By way of example, this data may include: message content, client device information, geolocation information, media annotations and overlays, message content persistence conditions, social network information, texture maps, virtual effects, and live event information. Data exchange within the messaging system 100 is invoked and controlled through functionality available through the user interface (UI) of the messaging client applications 104.

[0032] Turning now specifically to the messaging server system 108, an application program interface (API) server 110 is coupled to and provides a programming interface to an application server 112. The application server 112 is communicatively coupled to a database server 118, which facilitates access to a database 120 in which data associated with messages processed by the application server 112 is stored.

[0033] The application program interface (API) server 110 receives and sends message data (e.g., commands and message payloads) between the client device 102 and the application server 112. In particular, the application program interface (API) server 110 provides a set of interfaces (e.g., routines and protocols) that the messaging client application 104 can call or query to invoke functionality of the application server 112. The application program interface (API) server 110 exposes various functions supported by the application server 112, including: account registration, login functionality, sending messages from a particular messaging client application 104 to another messaging client application 104 via the application server 112, sending media files (e.g., images or videos) from the messaging client application 104 to the messaging server application 114, and possible access by another messaging client application 104, setting up collections of media data (e.g., stories), retrieving a friend list of a user of the client device 102, retrieving such collections, retrieving messages and content, adding friends to and removing friends from the social graph, the location of friends within the social graph, and open application events (e.g., related to the messaging client application 104).

[0034] The application server 112 hosts multiple applications and subsystems, including a messaging server application 114, a location sharing system 116, and a social networking system 122. The messaging server application 114 implements multiple message processing techniques and functions, particularly relating to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client application 104. As will be described in further detail, text and media content from multiple sources can be aggregated into content collections (e.g., referred to as stories or galleries). The messaging server application 114 then makes these collections available to the messaging client applications 104. Given the hardware requirements for other processor- and memory-intensive processing of data, such processing can also be performed on the server side by the messaging server application 114.

[0035] The application server 112 also includes a location sharing system 116 that is dedicated to performing various image processing operations, typically with respect to images or videos received within the payload of messages at the messaging server application 114 .

[0036] The application server 112 is communicatively coupled to a database server 118 , which facilitates access to a database 120 in which data associated with messages processed by the messaging server application 114 is stored.

[0037] The social networking system 122 supports various social networking functional services and makes these functional services available to the messaging server application 114. To this end, the social networking system 122 maintains and accesses an entity graph 204 (e.g., Figure 2 Examples of functionality and services supported by the social networking system 122 include identifying other users in the messaging system 100 who have a relationship with or are "following" a particular user and identifying other entities and interests of a particular user.

[0038] Examples of the functionality and services supported by the social networking system 122 also include generating a geographic-based graphical user interface (GUI). This interface may be referred to herein as a "graph GUI" and may be used in conjunction with a social media application. In some implementations, the graph GUI may include a representation of at least approximately the respective locations of a user and the user's friends in a social network graph accessed by the social media application using an avatar for each respective user.

[0039] Social networking system 122 may receive authorization from the user to use or refrain from using the user's location information. In some embodiments, social networking system 122 may also choose to share or not share the user's location with other users via the GUI. In some cases, regardless of whether the user shares his or her location with other users, the user's avatar may be displayed to the user on the display screen of the user's computing device.

[0040] In some embodiments, a user can turn off or on location sharing of the user from within the map GUI (e.g., via a setting accessed in conjunction with a menu presented in the map GUI). In some embodiments, social-networking system 122 may still present the user's avatar at the user's current location on the map GUI on the user's own device after the user has turned off location sharing. This mode is referred to herein as "ghost mode." In some embodiments, social-networking system 122 may present an icon on the display screen of the user's computing device to indicate that the user's location is not currently being shared with other users.

[0041] Note that the illusion mode functionality described herein can be distinguished from turning off location services on a mobile user device. Thus, in some embodiments, when illusion mode is on, the device location services are still running so that the user's location can still be determined.

[0042] In some embodiments, when a user turns on ghost mode after previously sharing their location and the user's avatar is displayed on the map, the user's avatar disappears from the map of other users. In some embodiments, while in ghost mode, the user can still see on the map anyone who has chosen to share their location with the user. In some embodiments, the user can also be provided with the option to specify who will see their location and at what granularity. Examples of granularity options that can be selected by the user include an "exact" option (e.g., the user's location will be presented on the map as accurately as the location information from the user's computing device can provide); and a random location within a predetermined area (e.g., a city) based on the location information from the user's computing device.

[0043] In some embodiments, when a user (or group of users) selects the random location granularity option, the user's avatar will be displayed in the map GUI within a predetermined distance of the user's current location (e.g., such as within a predetermined area of ​​the city the user is located in), and the position of the user's avatar will not change unless the user leaves the area. In some embodiments, the user's avatar may include a label specifying the geographic area the user is located in (e.g., "New York City").

[0044] In some implementations, a user can select a group of other users to whom their location will be displayed, and can specify different display attributes for different groups or individuals. In one example, the audience options include: "Best Friends," "Friends," and "Custom" (which is a whitelist at the individual level). In this example, if "Friends" is selected, all new people added to the user's friend list will automatically be able to see their location. If they have already shared with the user, their avatar will appear on the user's map.

[0045] In some embodiments, when viewing the map GUI, the user can see on the map the locations of all of his / her friends who have shared their location with the user, with each friend represented by their respective avatar. In some embodiments, if a friend does not have an avatar, a profile picture or a default icon displayed at the friend's corresponding location can be used to represent the friend.

[0046] In some embodiments, a user can select between friends on the graph via a menu, such as a carousel. In some embodiments, selecting a particular friend automatically focuses the graph view on that friend's avatar. Embodiments of the present disclosure may also allow a user to take various actions with the user's friends from within the graph GUI. For example, the system may allow the user to chat with the user's friends without leaving the graph. In one specific example, a user can select a chat icon from a menu presented in conjunction with the graph GUI to initiate a chat session.

[0047] Figure 2 is a schematic diagram illustrating a data structure 200 that may be stored in the database 120 of the messaging server system 108 according to certain example embodiments. Although the contents of the database 120 are shown as including a number of tables, it should be understood that the data may be stored in other types of data structures (e.g., as an object-oriented database).

[0048] The database 120 includes message data stored in a message table 208. The entity table 202 stores entity data, including an entity graph 204. The entities for which records are maintained in the entity table 202 may include individuals (e.g., users), corporate entities, organizations, objects, places, events, etc. Regardless of type, any entity for which the messaging server system 108 stores data may be an identified entity. Each entity is provided with a unique identifier and an entity type identifier (not shown). In addition, the entity graph 204 stores information about relationships and associations between entities. Such relationships may be, for example, simply social, professional (e.g., working at a common company or organization), interest-based, or activity-based. The location table 206 stores historical and current location information of the user (e.g., geographic location information determined by the GPS unit of the client device 102).

[0049] Now turn Figure 3 , a graphical representation of a processing environment 300 is shown, which includes at least a processor 302 (e.g., a GPU, a CPU, or a combination thereof). The processing environment 300 can be implemented in a user device such as the client device 102, which is arranged to capture video frames in real time and process and display an augmented or virtual reality 3D experience, as described below.

[0050] The processor 302 is shown coupled to a power source 304 and includes modules (permanently configured or temporarily instantiated), namely a location component 308, a GUI component 310, a messaging UI component 312, and a virtual effects UI component 314. The location component 308 is operable to determine the user's location based on the location information. The GUI component 310 is operable to generate a user interface and cause the user interface to be displayed on the client device. The messaging UI component 312 is operable to generate a user interface and cause the user interface to be displayed on the client device. As shown, the processor 302 can be communicatively coupled to another processor 306.

[0051] In some embodiments, the virtual effects UI component 314 performs semantic segmentation on the image frames (i.e., video stream) from the image capture device, as described in detail below, and generates an augmented or virtual reality 3D experience for presentation in the user interface generated by the GUI component 310. In some embodiments, the virtual effects UI component 314 is implemented as a graphics processing unit (GPU). In some embodiments, the processor 302 itself is a GPU.

[0052] Figure 4 4 is a block diagram 400 illustrating a software architecture 404 that can be installed on any one or more of the devices described herein. The software architecture 404 is supported by hardware such as a machine 402 including a processor 420, a memory 426, and I / O components 438. In this example, the software architecture 404 can be conceptualized as a stack of layers, where each layer provides specific functionality. The software architecture 404 includes various layers such as an operating system 412, libraries 410, a framework 408, and applications 406. In operation, the applications 406 invoke API calls 450 through the software stack and receive messages 452 in response to the API calls 450.

[0053] The operating system 412 manages hardware resources and provides common services. The operating system 412 includes, for example, a kernel 414, services 416, and drivers 422. The kernel 414 acts as an abstraction layer between the hardware layer and other software layers. For example, the kernel 414 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. Services 416 can provide other common services to other software layers. Drivers 422 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 422 may include display drivers, camera drivers, Bluetooth or Bluetooth low energy drivers, flash memory drivers, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), Driver, audio driver, power management driver, etc.

[0054] The library 410 provides a low-level common infrastructure used by the application 406. The library 410 may include a system library 418 (e.g., a C standard library) that provides functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the library 410 may include an API library 424, such as a media library (e.g., a library for supporting the presentation and manipulation of various media formats such as Moving Picture Experts Group 4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer 3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework for rendering graphical content on a display in two dimensions (2D) and three dimensions (3D), a database library (e.g., SQLite providing various relational database functions), a web library (e.g., WebKit providing web browsing functions), etc. The library 410 may also include a variety of other libraries 428 to provide many other APIs to the application 406.

[0055] The framework 408 provides a high-level common infrastructure used by the applications 406. For example, the framework 408 provides various graphical user interface (GUI) functions, advanced resource management, and advanced location services. The framework 408 can provide a wide range of other APIs that can be used by the applications 406, some of which may be specific to a particular operating system or platform.

[0056] In an example embodiment, applications 406 may include a home application 436, a contacts application 430, a browser application 432, a book reader application 434, a location application 442, a media application 444, a messaging application 446, a game application 448, and a variety of other applications such as third-party applications 440. Applications 406 are programs that perform functions defined in the program. Various programming languages ​​may be used to create one or more applications 406 structured in various ways, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In a specific example, third-party applications 440 (e.g., those developed by entities other than the vendor of a particular platform using ANDROID) may be used to create one or more applications 406. TM or IOS TM Software Development Kit (SDK) can be used to develop applications on platforms such as IOS TM ANDROID TM 、 Mobile software running on the mobile operating system of the phone or other mobile operating system. In this example, the third party application 440 can call the API call 450 provided by the operating system 412 to facilitate the functions described herein.

[0057] Figure 5 5 is a diagrammatic representation of a machine 500 in which instructions 508 (e.g., software, programs, applications, applet, application program, or other executable code) may be executed that cause the machine 500 to perform any one or more of the methodologies discussed herein. For example, the instructions 508 may cause the machine 500 to perform any one or more of the methodologies described herein. The instructions 508 convert a general-purpose, unprogrammed machine 500 into a specific machine 500 that is programmed to perform the functions described and illustrated in the manner described. The machine 500 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 500 may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 500 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a PDA, an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web device, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing instructions 508 specifying actions to be taken by the machine 500. Furthermore, while only a single machine 500 is shown, the term "machine" should also be construed to include a collection of machines that individually or collectively execute instructions 508 to perform any one or more of the methodologies discussed herein.

[0058] The machine 500 may include a processor 502, a memory 504, and an I / O component 542 that may be configured to communicate with each other via a bus 544. In an example embodiment, the processor 502 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), other processors, or any suitable combination thereof) may include, for example, a processor 506 that executes instructions 508 and a processor 510. The term "processor" is intended to include a multi-core processor that may include two or more independent processors (sometimes referred to as "cores") that may execute instructions concurrently. Although Figure 5Multiple processors 502 are shown, but the machine 500 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof. The processor 502 may be formed as Figure 3 The processing environment shown.

[0059] The memory 504 includes a main memory 512, a static memory 514, and a storage unit 516, all of which are accessible by the processor 502 via the bus 544. The main memory 504, the static memory 514, and the storage unit 516 store instructions 508 for implementing any one or more of the methodologies or functions described herein. The instructions 508 may also reside, completely or partially, within the main memory 512, within the static memory 514, within the machine-readable medium 518 within the storage unit 516, within at least one of the processors 502 (e.g., within a cache memory of the processor), or within any suitable combination thereof during execution thereof by the machine 500.

[0060] The I / O components 542 may include various components for receiving input, providing output, generating output, sending information, exchanging information, capturing measurements, etc. The specific I / O components 542 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It will be understood that the I / O components 542 may include Figure 5 Many other components are not shown in the drawings. In various example embodiments, the I / O components 542 may include output components 528 and input components 530. The output components 528 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. The input components 530 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens that provide location and / or force of touches or touch gestures, or other tactile input components), audio input components (e.g., microphones), video input components (e.g., cameras or other image capture devices), etc.

[0061] In other example embodiments, the I / O component 542 may include a biometric component 532, a motion component 534, an environmental component 536, or a positioning component 538, as well as various other components. For example, the biometric component 532 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), etc. The motion component 534 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The environment component 536 includes, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers for detecting ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones for detecting background noise), a proximity sensor component (e.g., an infrared sensor for detecting nearby objects), a gas sensor (e.g., a gas detection sensor for detecting concentrations of hazardous gases for safety or for measuring pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. The positioning component 538 includes a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer for detecting air pressure, from which altitude can be derived), an orientation sensor component (e.g., a magnetometer), etc.

[0062] A variety of technologies can be used to implement communications. I / O components 542 also include communication components 540 that are operable to couple machine 500 to network 520 or device 522 via coupling 524 and coupling 526, respectively. For example, communication components 540 may include a network interface component or other suitable device that interfaces with network 520. In another example, communication components 540 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, a Bluetooth component (e.g., Bluetooth Low Energy), Device 522 may be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via USB).

[0063] In addition, the communication component 540 can detect an identifier or include a component operable to detect an identifier. For example, the communication component 540 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting the following: a one-dimensional barcode, such as a universal product code (UPC) barcode; a multi-dimensional barcode, such as a Quick Response (QR) code, an Aztec code, a data matrix, a data glyph, a MaxiCode, a PDF417, a Hypercode, a UCC RSS-2D barcode, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be obtained via the communication component 540, such as a location via Internet Protocol (IP) geolocation, a location via Internet Protocol (IP), ... Location of signal triangulation, location of NFC beacon signals that can indicate a specific location via detection, etc.

[0064] Various memories (e.g., memory 504, main memory 512, static memory 514, and / or memory of processor 502) and / or storage unit 516 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 508), when executed by processor 502, cause various operations to implement the disclosed embodiments.

[0065] Instructions 508 may be sent or received over network 520 via a network interface device (e.g., a network interface component included in communications component 540) using a transmission medium and using any of a number of well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instructions 508 may be sent or received to device 522 via coupling 526 (e.g., a peer-to-peer coupling) using a transmission medium.

[0066] Figure 6 The main operational blocks in a routine 600 for performing semantic segmentation of an image frame for generating an augmented or virtual reality 3D experience are shown. The routine 600 may be executed by one or more processors of a client device, such as Figure 3 or Figure 5 The processor shown in performs.

[0067] In block 602 , a processor processes a plurality of image frames obtained from at least one image capture device to build a world space map.

[0068] Note that an image frame may be the output of more than one image capture device. In some cases, multiple image capture devices may be integrated with the same client device (e.g., front and rear cameras, multiple cameras provided for depth of field and / or three-dimensional image capture), or they may be provided in separate client devices and communicated to the processor (e.g., via a communication network or messaging system such as Figure 1 ).

[0069] In block 604 , for at least one image frame of the plurality of image frames, the processor determines a segmentation.

[0070] In block 606 , the processor generates user interface data for display using the world space map and the segmentation.

[0071] The processor may optionally initialize a segmentation texture for storing segmentation information before determining the segmentation at block 604. The segmentation determined at block 604 is then stored in the initialized segmentation texture. The segmentation information may, for example, relate to a fixed area around the at least one image capturing device.

[0072] In some embodiments, building a world space map at block 602 is performed by applying a surface tracker to multiple image frames. A surface tracker is a software component capable of tracking a gravity-aligned plane. Examples of suitable surface tracker components include, but are not limited to, full Simultaneous Localization and Mapping (SLAM) systems such as ARKit and ARCore. For the purpose of tracking simple surfaces such as the ground (or floor), the surface tracker only needs to be arranged to track a gravity-aligned plane: the client device will need an accelerometer or some other equivalent component to determine the direction of gravity. In some embodiments (e.g., embodiments where it is desired to segment image frames into "ground / floor" and "non-ground / non-floor"), the surface tracker component can simply be a plane tracker, and such a simple surface tracker can be referred to as a "lightweight tracker." Therefore, compared to techniques that require full SLAM, the present disclosure places lower requirements on the specifications of the accelerometers and gyroscopes (e.g., inertial measurement units, IMUs) used by the client device to monitor the movement and orientation of the client device.

[0073] To accurately generate a world space map, the surface tracker requires more than one image frame (in addition to the accelerometer / gyroscope data for gravity alignment).

[0074] In some embodiments, determining the segmentation (block 604) includes: obtaining a segmentation in screen space from at least one image frame of the plurality of image frames; projecting the screen space segmentation into world space; and integrating the projected segmentation into a world space map using a temporal filter (see below). Figure 8).

[0075] In some embodiments, the segmentation in screen space is obtained by executing a semantic segmentation neural network with at least one image frame from the plurality of image frames as input. Notably, the segmentation need not be obtained for every image frame. Thus, the determination of the segmentation can be iteratively repeated for subsets of the plurality of image frames.

[0076] In some embodiments, the method further includes: determining whether a sufficient condition is satisfied; and if the sufficient condition is satisfied, stopping execution of the semantic segmentation neural network and continuing to use the determined segmentation in subsequent iterations.

[0077] An exemplary condition for stopping processing may be when the variance of a pixel is less than a variance threshold, where "variance" is a measure of the average difference in the probability that the pixel is correctly labeled.

[0078] In certain embodiments, the method may be arranged to be spatially selective with respect to the region or regions of the image frame where execution of the semantic segmentation neural network is to be stopped.

[0079] Determining the segmentation may also include storing the screen space segmentation in a segmentation texture.

[0080] Figure 7 The embodiment of generating user interface data for display (such as Figure 6 The main operation block in the task of block 606).

[0081] In block 702 , the processor optionally generates real-time virtual effects in screen space using a world space map and segmentation.

[0082] In block 704, the processor then optionally displays the real-time virtual effect in the current image.

[0083] Displaying the effect in the current image may include overlaying the virtual effect on the current image or replacing pixels of the current image with alternative pixels or shaders from the real-time virtual effect.

[0084] Figure 8 shows that in implementing the determination of segmentation (such as Figure 6 The main operation block in the task of block 604).

[0085] In block 802, a processor processes a plurality of image frames obtained from at least one image capture device to create a world-space map. In block 804, the processor determines a segmentation for at least one of the plurality of image frames. In block 806, the processor uses the world-space map and the segmentation to generate user interface data for display. In block 808, the processor obtains a segmentation in screen space from at least one of the plurality of image frames. In block 810, the processor projects the screen-space segmentation into world space. In block 812, the processor integrates the projected segmentation into the world-space map using a temporal filter.

[0086] In some embodiments, the screen space segmentation is projected into world space by storing the projected segmentation in a temporary world space segmentation texture and then integrating the temporary world space segmentation texture from the frame into the world space map. While the use of a temporary world space segmentation texture provides an efficient data structure in client devices where processing power is more important than storage (and retrieval) latency, the use of a temporary world space segmentation texture is optional in client devices with more advanced processing facilities.

[0087] In some embodiments, the segmentation texture is a two-dimensional data structure, where each entry in the segmentation texture includes at least one value indicating the likelihood that a corresponding pixel in the image frame satisfies a segmentation criterion. In the case of ground / floor segmentation, the segmentation texture is a ground / floor segmentation texture, and the segmentation criterion can be a criterion for assigning pixels to a ground / floor object class. The ground / floor segmentation texture is essentially a 2D map with corresponding "color values" indicating "floor" or "non-floor" segmentation.

[0088] The temporal filter may be an infinite impulse response (IIR) filter or a finite impulse response (FIR) filter.

[0089] like Figure 3 As noted in the discussion of , the processor implementing the above method may be a graphics processing unit (GPU). Notably, the above method comprises a series of highly parallelizable steps and may be implemented as pixel / fragment shaders; tasks for which the GPU is well suited.

[0090] In some embodiments, the method further comprises running at least one world-space post-processing filter. Post-processing can be applied to the world-space graph because this provides visually stable results in the resulting display of the real-time virtual effect. Examples of world-space post-processing filters include erosion filters and dilation filters.

[0091] In one use case, the segmentation method described above is used to replace pixels depicting a static floor with a virtually modified floor effect in a generated UI video display. Figure 9A "ground truth" street scene with parked vehicles and surrounding shops is shown.

[0092] exist Figure 10 In the above segmentation method, the static road surface ( Figure 9 The pixels in the image above are replaced with a virtually modified ground effect. The ground in the resulting video display can be replaced with an effect where the road surface is textured to resemble lava. In contrast, the parked vehicles and surrounding shops remain unchanged.

[0093] Figure 9 The road surface in the example may alternatively be replaced by a surface in which the height of the floor is dynamically changed to depict an undulating water surface (e.g. Figure 11 shown).

[0094] In a further use case, the segmentation method described above can be used to generate a floor plan of a room. By panning the image capture device around the room to generate a set of image frames that comprehensively span the entire room (including furniture, fixtures, and accessories), a routine for segmenting the image frames enables rapid identification of exposed floor space, and the resulting world-space model can be used for architectural, structural surveying, and / or interior design applications.

[0095] In a similar use case, the segmentation method described above can be used to generate a map of an outdoor area, allowing one to quickly determine where buildings are located (via their footprints in the segmented world-space map of the ground).

[0096] The generated world-space graph can also be used to help a virtual agent (e.g., an AI character / "sprite" displayed as walking on the floor in an AR game) navigate a mixed reality environment based on real-time video capture. Similarly, the presence of walls and other obstacles can be modeled using information derived from the generated world-space graph so that a virtual agent modeled as moving on the floor can avoid (or cause) collisions with obstacles in the modeled behavior. Thus, for example, a virtual remote-controlled car can "drive" around a room (in augmented reality) and collide with walls and other objects.

[0097] The generated world space map can also be compared with a pre-existing location map (e.g., a known floor plan or a known street map) to help accurately orient the user relative to the pre-existing map. This can be particularly useful in indoor or outdoor areas (downtown neighborhoods, shopping malls, etc.) where alternative positioning technologies (e.g., GPS, Cell-ID, etc.) are insufficient. The pre-existing location map and the location on the map inferred from the comparison can be presented in a map GUI (e.g., the map GUI discussed above).

[0098] In the above discussion, the term screen space is intended to include the nearly equivalent case of “camera space.” Camera space refers to the data obtained by the image capture device in any given image frame, which may be cropped for display to give screen space.

[0099] In some embodiments, a semantic segmentation neural network can be used to segment an image frame in screen space into more than two labeled categories. For example, an image frame can be segmented into "floor / ground," "road markings," and "neither floor nor road markings." Different real-time virtual effects can be generated for each labeled category.

[0100] In some use cases, the segmentation method described above can be used to identify unclassified portions of the world space map, for example, portions that are not classified as "floor / ground". In some cases, the unclassified portions of the image frame may be replaced with a virtually modified "wall" effect in the generated UI video display, such that a virtual wall may be applied at the location of obstacles on the floor, and virtual objects may be occluded when rendered as moving "behind" the virtual wall. This can be achieved by virtually extrapolating a vertical pixel stack (which can conveniently be an infinite stack) from the pixels marked as non-floor on the ground plane and using this generated virtual surface as an occluder when rendering other virtual objects in the generated UI video display.

[0101] In some embodiments, a messaging system such as Figure 1 The messaging system outlined in is used to communicate information about the world-space graph, including the projected segmentation integrated into the world-space graph, so that more than one client device can access (and contribute to) a shared world view graph (and segmentation texture).

[0102] In some embodiments, the segmentation method described above can be adapted to identify movable objects (e.g., people, vehicles, etc.) that temporarily occlude the floor. Sufficient image frames can be processed to track the "real" floor surface and derive a "cleaned" world-space map that effectively extracts the temporary footprints of such movable objects.

[0103] Similarly, certain embodiments facilitate identifying and tracking regions of an image frame that share similar color values ​​(sometimes referred to as "blobs"). The blobs can be labeled using the segmentation method above, and a visual effect can be produced in which the "blob" segmentation can be removed: this can be used to anonymize or blur features in a world space map.

Claims

1. A computer-implemented method comprising: processing a plurality of image frames obtained from at least one image capture device to create a world space map; determining a segmentation of a first image frame of the plurality of image frames; iteratively repeating the determination of the segmentation of each image frame subsequent to the first image frame until it is determined that a sufficient condition is satisfied; based on determining that the sufficient condition is satisfied, stopping the execution of the determination of the segmentation for each image frame subsequent to the first image frame, and continuing to use the last determined segmentation in subsequent iterations of image frames in the plurality of image frames; as well as User interface data is generated for display using the world space map and the segmentation.

2. The computer-implemented method of claim 1 , wherein: Generating user interface data for display includes: generating real-time virtual effects in screen space using the world-space map and the segmentation; and The real-time virtual effect is displayed in the current image.

3. The computer-implemented method of claim 1 , further comprising: A segmentation texture for storing segmentation information is initialized before determining the segmentation.

4. The computer-implemented method of claim 3, wherein: The segmentation information relates to a fixed area around the at least one image capturing device.

5. The computer-implemented method of claim 1 , wherein: The world space map is built by applying a surface tracker to the plurality of image frames.

6. The computer-implemented method of claim 1 , wherein: Determining the segmentation includes: obtaining a segmentation in screen space from the at least one image frame of the plurality of image frames; Projecting the screen space divisions into world space; and The projected segmentation is integrated into the world space map using a temporal filter.

7. The computer-implemented method of claim 6, wherein: The segmentation in screen space is obtained by executing a semantic segmentation neural network with the at least one image frame of the plurality of image frames as input.

8. The computer-implemented method of claim 6, wherein: Determining the segmentation further comprises: The screen space segmentation is stored in a segmentation texture.

9. The computer-implemented method of claim 6, wherein: The screen space segmentation is projected into world space by storing the projected segmentation in a temporary world space segmentation texture and then integrating the temporary world space segmentation texture from the frame into the world space map.

10. The computer-implemented method of claim 3, wherein: The segmentation texture is a two-dimensional data structure, and each entry in the segmentation texture includes at least one value indicating a likelihood that a corresponding pixel in the image frame satisfies a segmentation criterion.

11. The computer-implemented method of claim 10, wherein: The segmentation texture is a ground / floor segmentation texture, and the segmentation criterion is a criterion for assigning pixels to a ground / floor object class.

12. The computer-implemented method of claim 6, wherein: The temporal filter is an infinite impulse response filter or a finite impulse response filter.

13. The computer-implemented method of claim 1 , further comprising: Run at least one world-space post-processing filter.

14. A computing device, comprising: processor; as well as a memory storing instructions that, when executed by the processor, configure the computing device to: processing a plurality of image frames obtained from at least one image capture device to create a world space map; determining a segmentation of a first image frame of the plurality of image frames; iteratively repeating the determination of the segmentation of each image frame subsequent to the first image frame until it is determined that a sufficient condition is satisfied; based on determining that the sufficient condition is satisfied, stopping the execution of the determination of the segmentation for each image frame subsequent to the first image frame, and continuing to use the last determined segmentation in subsequent iterations of image frames in the plurality of image frames; as well as User interface data is generated for display using the world space map and the segmentation.

15. The computing device of claim 14, wherein: Generating user interface data for display includes: generating real-time virtual effects in screen space using the world-space map and the segmentation; and The real-time virtual effect is displayed in the current image.

16. The computing device of claim 14, wherein: The instructions further configure the computing device to: A segmentation texture for storing segmentation information is initialized before determining the segmentation.

17. The computing device of claim 14, wherein: The world space map is built by applying a surface tracker to the plurality of image frames.

18. The computing device of claim 14, wherein: Determining the segmentation includes: obtaining a segmentation in screen space from the at least one image frame of the plurality of image frames; Projecting the screen space divisions into world space; and The projected segmentation is integrated into the world space map using a temporal filter.

19. A computer-readable storage medium carrying instructions that, when executed by a computer, cause the computer to perform operations comprising: processing a plurality of image frames obtained from at least one image capture device to create a world space map; determining a segmentation of a first image frame of the plurality of image frames; iteratively repeating the determination of the segmentation of each image frame subsequent to the first image frame until it is determined that a sufficient condition is satisfied; based on determining that the sufficient condition is satisfied, stopping the execution of the determination of the segmentation for each image frame subsequent to the first image frame, and continuing to use the last determined segmentation in subsequent iterations of image frames in the plurality of image frames; as well as User interface data is generated for display using the world space map and the segmentation.

Images