Dynamic contextual media filter

Abstract

A context filter system configured to perform operations including capturing an image frame on a client device—the image frame includes a description of an object—identifying an object category of the object based on the description of the object within the image frame, accessing media content associated with the object category within a media storage, generating a presentation of the media content, and causing a display of the presentation of the media content within the image frame on the client device.

Description

Technology Field

[0001] The embodiments of the present disclosure generally relate to mobile computing technology, and, more specifically, to systems for generating and displaying dynamic contextual media content on a client device, without limitation. Background Technology

[0002] Augmented reality (AR) is a live, direct, or indirect view of a physical real-world environment that has elements augmented by computer-generated sensory inputs. Brief explanation of the drawing

[0003] To easily identify the discussion of any specific element or action, the top digit or numbers of the reference numbers refer to the drawing number where the element is first introduced. FIG. 1 is a block diagram illustrating an exemplary messaging system for exchanging data (e.g., messages and associated content) through a network according to some embodiments, wherein the messaging system includes a contextual filter system. FIG. 2 is a block diagram illustrating additional details regarding a messaging system according to exemplary embodiments. FIG. 3 is a block diagram illustrating various modules of a context filter system according to specific exemplary embodiments. FIG. 4 is a flowchart illustrating a method for causing a display of a context filter according to specific exemplary embodiments. FIG. 5 is a flowchart illustrating a method for causing a display of a context filter according to specific exemplary embodiments. FIG. 6 is a flowchart illustrating a method for creating a context filter to be displayed on a client device according to certain exemplary embodiments. FIG. 7 is an interface flow diagram depicting interfaces presented by a context filter system according to certain exemplary embodiments. FIG. 8 is a block diagram illustrating a representative software architecture that can be used to implement various embodiments and can be used with various hardware architectures described in this specification. FIG. 9 is a block diagram illustrating components of a machine according to some exemplary embodiments capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and performing one or more of the methodologies discussed herein. Specific details for implementing the invention

[0004] As discussed above, AR systems provide users with graphical user interfaces (GUIs) to display live direct or indirect views of a physical real-world environment, wherein elements of the view are augmented by computer-generated sensory inputs. For example, an AR interface can present media content at locations within the display of the view of the real-world environment so that the media content appears to interact with elements in the real-world environment.

[0005] The exemplary embodiments described herein relate to a context filter system for generating and displaying context filters on a client device. According to specific embodiments, the context filter system is configured to perform operations including capturing an image frame on a client device—the image frame comprising a depiction of an object—identifying an object category of the object based on the depiction of the object within the image frame, accessing media content associated with the object category in a media storage, generating a presentation of the media content, and causing a display of the presentation of the media content within the image frame on the client device.

[0006] The media content may include animated Graphics Interchange Format (GIF) images of various shapes, sizes, and themes. In some embodiments, the context filter system may communicate with a media repository containing a sorted and labeled collection of the media content, wherein the media content within the collection is labeled or tagged based on the attributes of the media content. For example, the media content may be tagged with a label identifying an object category of the media content, such as "food" or "basketball," and thus a reference to the object category corresponds to a set of media content within the collection of media content.

[0007] In some embodiments, in response to receiving input selecting a user option from a set of user options, the context filter system captures an image from a client device, wherein the image frame contains a depiction of one or more objects in a real-world environment and identifies object categories of one or more of the objects depicted within the image frame. For example, the context filter system may detect one or more Quick Response (QR) codes within the image frame—wherein the QR codes identify the object depicted within the image frame or the object category associated with the object—or, in additional embodiments, may identify the object depicted within the image frame using one or more image and text recognition techniques. Based on the identification of the object or object category, the context filter system searches for a set of labels or tags associated with the object or object category and queries a media repository to identify media content based on this set of labels or tags. Thus, the identified media content from the content repository may include a set of media content associated with the object or object category identified within the image frame.

[0008] In response to identifying the set of media content from the media storage, the context filter system generates the set of media content on the client device and causes its display. In some embodiments, to generate a presentation of the set of media content, the context filter system retrieves a media template that provides a presentation format and layout for the set of media content. For example, the media template may define locations for presenting the set of media content within an image frame on the client device. In some embodiments, the presentation of the set of media content within the image frame on the client device may be based on the locations (or locations) of the objects identified within the image frame.

[0009] FIG. 1 is a block diagram illustrating an exemplary messaging system (100) for exchanging data (e.g., messages and associated content) over a network. The messaging system (100) includes one or more client devices (102) that host a plurality of applications, each including a messaging client application (104). Each messaging client application (104) is coupled to communicate with other instances of the messaging client application (104) and the messaging server system (108) over a network (106) (e.g., the Internet).

[0010] Accordingly, each messaging client application (104) can communicate with other messaging client applications (104) and a messaging server system (108) and exchange data through a network (106). The data exchanged between messaging client applications (104) and between a messaging client application (104) and a messaging server system (108) includes functions (e.g., commands to activate functions) as well as payload data (e.g., text, audio, video, or other multimedia data).

[0011] The messaging server system (108) provides server-side functionality to a specific messaging client application (104) via the network (106). Although specific functions of the messaging system (100) are described in this specification as being performed by the messaging client application (104) or by the messaging server system (108), it will be recognized that the location of specific functions within the messaging client application (104) or the messaging server system (108) is a design choice. For example, it may be technically desirable to initially place specific technologies and functions within the messaging server system (108), but later transfer these technologies and functions to the messaging client application (104) when the client device (102) has sufficient processing capacity.

[0012] The messaging server system (108) supports various services and operations provided to the messaging client application (104). Such operations include transmitting data to the messaging client application (104), receiving data from it, and processing data generated by it. In some embodiments, this data includes, for example, message content, client device information, geolocation information, media annotations and overlays, message content persistence conditions, social network information, and live event information. In other embodiments, other data is used. Data exchange within the messaging system (100) is initiated and controlled through functions available via the GUIs of the messaging client application (104).

[0013] Now, referring specifically to the messaging server system (108), an application program interface (API) server (110) is coupled to an application server (112) to provide a programmatic interface. The application server (112) is communicably coupled to a database server (118), which facilitates access to a database (120) where data associated with messages processed by the application server (112) is stored.

[0014] Specifically, regarding the application program interface (API) server (110), this server receives and transmits message data (e.g., commands and message payloads) between the client device (102) and the application server (112). Specifically, the application program interface (API) server (110) provides a set of interfaces (e.g., routines and protocols) that can be called or queried by a messaging client application (104) to activate the functionality of the application server (112). The application program interface (API) server (110) exposes various functions supported by the application server (112), including account registration, login functionality, transmission of messages via the application server (112) from a specific messaging client application (104) to another messaging client application (104), transmission of media files (e.g., images or videos) from a messaging client application (104) to a messaging server application (114), and possible access by another messaging client application (104), such as setting up a collection of media data (e.g., stories), searching for a list of friends of a user of a client device (102), searching for such collections, searching for messages and content, adding and removing friends to a social graph, the location of friends within a social graph, and opening application events (e.g., related to the messaging client application (104)).

[0015] The application server (112) hosts a number of applications and subsystems, including a messaging server application (114), an image processing system (116), a social network system (122), and a context filter system (124). The context filter system (124) is configured, according to some exemplary embodiments, to capture an image frame containing a description of an object from a client device (102), to identify one or more objects or object categories within the image frame based on the description of the object, to select one or more labels or tags based on the objects or object categories, to retrieve a set of media content based on the labels or tags, and to cause a display of a presentation of the set of media content within the image frame on the client device. Further details of the context filter system (124) may be found in FIG. 3 below.

[0016] The messaging server application (114) implements a number of message processing techniques and functions, particularly those related to the aggregation and other processing of content (e.g., text and multimedia content) contained in messages received from multiple instances of the messaging client application (104). As described in more detail, text and media content from multiple sources may be aggregated into collections of content (e.g., called stories or galleries). These collections are then made available to the messaging client application (104) by the messaging server application (114). Other processor and memory-intensive data processing may also be performed on the server side by the messaging server application (114), taking into account the hardware requirements for such processing.

[0017] The application server (112) also includes an image processing system (116) dedicated to performing various image processing operations on an image or video received within the payload of a message in a messaging server application (114).

[0018] The social network system (122) supports various social networking functions and services and enables the messaging server application (114) to use these functions and services. To this end, the social network system (122) maintains and accesses an entity graph (304) within a database (120). Examples of functions and services supported by the social network system (122) include the identification of other users of the messaging system (100) with whom a specific user has a relationship or "follows," and also the identification of other entities and the interests of the specific user.

[0019] The application server (112) is communicably coupled to the database server (118), which facilitates access to the database (120) where data associated with messages processed by the messaging server application (114) is stored.

[0020] FIG. 2 is a block diagram illustrating further details regarding a messaging system (100) according to exemplary embodiments. Specifically, the messaging system (100) is illustrated as including a messaging client application (104) and an application server (112), which ultimately implements a number of subsystems, namely a short-term timer system (202), a collection management system (204), and an annotation system (206).

[0021] The short-term timer system (202) is responsible for enforcing temporary access to content allowed by the messaging client application (104) and the messaging server application (114). To this end, the short-term timer system (202) includes a plurality of timers that selectively display messages and associated content through the messaging client application (104) and enable access thereto, based on duration and display parameters associated with messages, collections of messages (e.g., SNAPCHAT stories), or graphic elements. Further details regarding the operation of the short-term timer system (202) are provided below.

[0022] The collection management system (204) is responsible for managing collections of media (e.g., collections of text, image, video, and audio data). In some examples, collections of content (e.g., messages containing images, videos, text, and audio) may be organized into "event galleries" or "event stories." Such collections may be available for a specified period, such as the duration of the event to which the content is related. For example, content related to a music concert may be available as a "story" for the duration of that music concert. The collection management system (204) may also be responsible for posting an icon in the user interface of the messaging client application (104) that provides a notification of the existence of a specific collection.

[0023] The collection management system (204) further includes a curation interface (208) that allows a collection manager to manage and curate specific collections of content. For example, the curation interface (208) enables an event organizer to curate a collection of content related to a specific event (e.g., remove inappropriate content or duplicate messages). Additionally, the collection management system (204) automatically curates content collections using machine vision (or image recognition technology) and content rules. In certain embodiments, a reward may be paid to the user for including user-generated content in a collection. In such cases, the curation interface (208) operates to automatically pay such users for using their content.

[0024] The annotation system (206) provides various functions that enable the user to annotate media content associated with a message, or otherwise modify or edit it. For example, the annotation system (206) provides functions related to the creation and publication of media overlays for messages processed by the messaging system (100). The annotation system (206) effectively supplies media overlays (e.g., SNAPCHAT filters) to the messaging client application (104) based on the geolocation of the client device (102). In another example, the annotation system (206) effectively supplies media overlays to the messaging client application (104) based on other information, such as the social network information of the user of the client device (102). Media overlays may include audio and visual content and visual effects. Examples of audio and visual content include photos, texts, logos, animations, and sound effects, as well as animated face models such as those generated by the context filter system (124). Examples of visual effects include color overlays. Audio and visual content or visual effects may be applied to media content items (e.g., photos) on the client device (102). For example, a media overlay may include text that can be overlaid on a photo that is taken and generated by the client device (102). In another example, the media overlay may include an identification of a location overlay (e.g., Venice beach), the name of a live event, or the name of a merchant overlay (e.g., Beach Coffee House). In another example, the annotation system (206) uses the geolocation of the client device (102) to identify a media overlay containing the name of a merchant at the geolocation of the client device (102). The media overlay may include other indications associated with the merchant.Media overlays are stored in a database (120) and can be accessed through a database server (118).

[0025] In one exemplary embodiment, the annotation system (206) provides a user-based publication platform that enables users to select a geolocation on a map and upload content associated with the selected geolocation. Users can also specify situations in which a particular media overlay should be provided to other users. The annotation system (206) generates a media overlay that includes the uploaded content and associates the uploaded content with the selected geolocation.

[0026] In another exemplary embodiment, the annotation system (206) provides a merchant-based publishing platform that enables merchants to select a specific media overlay associated with a geolocation through a bidding process. For example, the annotation system (206) associates the media overlay of the highest bidder with a corresponding geolocation for a predefined amount of time.

[0027] FIG. 3 is a block diagram illustrating components of a context filter system (124) according to some exemplary embodiments, which configure the context filter system (124) to perform actions such as capturing an image frame containing a description of an object from a client device (102), identifying one or more objects or object categories within the image frame based on the description of the object, selecting one or more labels or tags based on the objects or object categories, searching for a set of media content based on the labels or tags, and causing a display of a presentation of the set of media content within the image frame on the client device.

[0028] The context filter system (124) is illustrated as including a presentation module (302), a media module (304), a communication module (306), and an identification module (308), all of which are configured to communicate with each other (e.g., via a bus, shared memory, or switch). Any one or more of these modules may be implemented using one or more processors (310) (e.g., by configuring such one or more processors to perform the functions described for the module) and thus may include one or more of the processors (310).

[0029] One or more of the described modules may be implemented using hardware alone (e.g., one or more of the processors (310) of the machine) or a combination of hardware and software. For example, any described module of the context filter system (124) may physically include an array of one or more processors (310) configured to perform the operations described herein for that module (e.g., one or more of the processors of the machine or a subset thereof). As another example, any module of the context filter system (124) may include software, hardware, or both that constitute an array of one or more processors (310) (e.g., one or more of the processors of the machine) to perform the operations described herein for that module. Thus, different modules of the context filter system (124) may include and be configured with different arrays of such processors (310) or a single array of such processors (310) at different times. Furthermore, any two or more modules of the context filter system (124) may be combined into a single module, and the functions described herein for the single module may be subdivided among multiple modules. Furthermore, according to various exemplary embodiments, the modules described herein as being implemented within a single machine, database, or device may be distributed across multiple machines, databases, or devices.

[0030] FIG. 4 is a flowchart illustrating a method (400) that causes a display of a context filter on a client device (102) according to certain exemplary embodiments. The operations of the method (400) may be performed by the modules described above with respect to FIG. 3. As illustrated in FIG. 4, the method (400) includes one or more operations (402, 404, 406, 408, and 410).

[0031] In operation 402, the presentation module (302) captures an image frame from the client device (102)—wherein the image frame includes a depiction of an object at a location within the image frame. For example, the presentation module (302) can activate the camera of the client device (102) and allow the camera of the client device (102) to capture the image frame.

[0032] In operation 404, the identification module (308) identifies an object depicted within an image frame. In some embodiments, to identify the object, the identification module (308) may perform one or more image or pattern recognition techniques using computer vision. In additional embodiments, the identification module (308) may identify one or more QR codes within an image frame and identify the object based on the QR codes.

[0033] In operation 406, the media module (304) accesses media content associated with an identified object within a media repository (e.g., databases (120)). For example, in some embodiments, the identification module (308) may select one or more tags or labels based on the identification of an object within an image frame, and cause the media module (304) to query the media repository based on the selected tags or labels. Thus, the media module (304) can access a set of media content associated with an object identified within an image frame by referencing media content labeled within the media repository with the selected tags or labels.

[0034] In operation 408, the presentation module (302) generates a presentation of a set of media content to be displayed within an image frame on the client device (102). For example, the presentation of a set of media content may include an AR display of the set of media content so that the retrieved media content appears to interact with elements depicted within the image frame in real time.

[0035] In some embodiments, to create a presentation of media content, the presentation module (302) retrieves a media template, wherein the media template defines a presentation format and layout to be applied to a set of media content. For example, the media template may define positions and orientations for presenting a set of media content within an image frame on a client device (102).

[0036] In some embodiments, to generate a presentation of media content to be displayed on a client device (102), the media module (304) provides the client device (102) with identifiers for each media content in a set of media content, thereby enabling the client device (102) to identify the relevant media content in a local memory storage.

[0037] In operation 410, the presentation module (302) causes the display of a presentation of a set of media content within an image frame on the client device (102). In some embodiments, the presentation of a set of media content within an image frame may be based on the location of an object identified within the image frame. For example, the presentation module (302) may orient the presentation of a set of media content within an image frame based on the location of an identified object.

[0038] In some embodiments, a user of the client device (102) may provide one or more inputs to generate a message based on an image frame containing a presentation of a set of media content, and may cause a communication module (308) to distribute the message to one or more recipients defined by the user of the client device (102).

[0039] FIG. 5 is a flowchart illustrating a method (500) for causing a display of a context filter on a client device (102) according to certain exemplary embodiments. The operations of the method (500) may be performed by the modules described above with respect to FIG. 3. As illustrated in FIG. 5, the method (500) includes one or more operations (502, 504, and 506).

[0040] In operation 502, in response to accessing media content in the media repository, the media module (304) retrieves a media template to create a presentation of a set of media content to be displayed within an image frame. In some embodiments, the media module (304) may access a template repository (e.g., a database (120)) to retrieve a media template corresponding to an object category associated with an object identified within the image frame.

[0041] In operation 504, the media module (304) creates a presentation of a set of media content by populating a media template with a set of media content accessed from a media repository.

[0042] In operation 506, the presentation module (302) causes the presentation of a filled template within an image frame to be displayed on the client device (102). In some embodiments, the presentation module (302) may display the filled template at a location within the image frame based on the location of an object identified within the image frame.

[0043] FIG. 6 is a flowchart illustrating a method (500) for generating a context filter to be displayed on a client device (102) according to certain exemplary embodiments. The operations of the method (600) may be performed by the modules described above with respect to FIG. 3. As illustrated in FIG. 6, the method (600) includes one or more operations (602, 604, and 606).

[0044] In operation 602, as in operation 402, the presentation module (302) captures an image frame from the client device (102)—wherein the image frame includes a depiction of an object at a location within the image frame. For example, the presentation module (302) may activate the camera of the client device (102) and allow the camera of the client device (102) to capture the image frame. In some embodiments, the image frame captured by the presentation module (302) may include image metadata including location data, time data, and device data of the client device (102).

[0045] In operation 604, the identification module (308) determines the context of the client device (102) in response to capturing an image frame. For example, the context may include the location of the client device, the time at which the image frame was captured, as well as the device type of the client device (102).

[0046] In some embodiments, the identification module (308) can parse the metadata of an image frame to determine relevant context information from location data, time data, and device data of the metadata.

[0047] In operation 608, the media module (304) accesses the media storage to identify related media content based on the identification of an object and the context of the client device (102). For example, media content may be tagged with location information or time information within the media storage, so that a reference to a specific time, season, day of the week, month, or place can identify a set of related media content.

[0048] FIG. 7 is an interface flow diagram (700) depicting interfaces presented by a context filter system (124) according to certain exemplary embodiments. The operations depicted by the interfaces of the flow diagram (700) can be performed by the modules described above with respect to FIG. 3.

[0049] The interface (705) describes an image frame captured by the client device (102). As illustrated in FIG. 7, the interface (705) includes a description of an object (720) at a location within the interface (705).

[0050] In some embodiments, the client device (102) can enable the context filter system (124) and cause the context filter system (124) to capture an image frame depicted in the interface (705) in response to receiving user input selecting a user option (725) as displayed in the interface (705).

[0051] In response to receiving input selecting a user option (725), the context filter system (124) may cause the display of a graphic icon (730) within the interface (710) to indicate that the context filter system (124) is activated.

[0052] The interface (715) includes a presentation of a set of media content (735) displayed within an image frame captured by the client device (102). As can be seen from the interface (715), the presentation of the set of media content (735) may be displayed at a location within the image frame based on the location of the object (720). As can be seen from the interface (715), the presentation of the set of media content (735) may include a plurality of media items including images and GIFs related to the object (720).

[0053] For example, as illustrated in FIG. 7, the object (720) is a chip bag. A context filter system (124) identifies the object category of the object (720) (e.g., food, snack, etc.) and searches for a set of media content (735), wherein the set of media content (735) includes media content tagged or labeled with the object category of the object (720).

[0054] The locations of each media content among the set of media content (735) within the image frame depicted in the interface (705) may be determined based on a media template, as discussed in methods (400, 500, and 600). By doing so, the user of the client device (102) may generate a message containing the set of media content (735) to be distributed to one or more recipients identified by the user of the client device (102). In some embodiments, the message may include a short-term message.

[0055] Software Architecture

[0056] FIG. 8 is a block diagram illustrating an exemplary software architecture (806) that may be used with various hardware architectures described herein. FIG. 8 is a non-limiting example of a software architecture, and it will be recognized that many other architectures may be implemented to facilitate the functionality described herein. The software architecture (806) may be executed on hardware such as the machine (900) of FIG. 9, which includes processors (904), memory (914), and I / O components (918), among other things. A representative hardware layer (852) is illustrated and may represent, for example, the machine (800) of FIG. 8. The representative hardware layer (852) includes a processing unit (854) having associated executable instructions (804). The executable instructions (804) represent executable instructions of the software architecture (806), which include implementations of methods, components, etc., described herein. The hardware layer (852) also includes memory / storage (856), which are memory and / or storage modules, and these also have executable instructions (804). The hardware layer (852) may also include other hardware (858).

[0057] In the exemplary architecture of FIG. 8, the software architecture (806) can be conceptualized as a stack of layers, each layer providing specific functionality. For example, the software architecture (806) may include layers such as an operating system (802), libraries (820), applications (816), and a presentation layer (814). During operation, applications (816) and / or other components within the layers may initiate application programming interface (API) API calls (808) through the software stack and receive responses in response to API calls (808). The layers illustrated are essentially representative, and not all software architectures have all layers. For example, some mobile or special-purpose operating systems may not provide frameworks / middleware (818), while others may provide such layers. Other software architectures may include additional or different layers.

[0058] The operating system (802) may manage hardware resources and provide common services. The operating system (802) may include, for example, a kernel (822), services (824), and drivers (826). The kernel (822) may serve as an abstraction layer between the hardware and other software layers. For example, the kernel (822) may be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. Services (824) may provide other common services for other software layers. Drivers (826) are responsible for controlling or interfacing with the underlying hardware. For example, drivers (826) include, depending on the hardware configuration, a display driver, a camera driver, a Bluetooth® driver, a flash memory driver, a serial communication driver (e.g., a Universal Serial Bus (USB) driver), a Wi-Fi® driver, an audio driver, a power management driver, etc.

[0059] Libraries (820) provide a common infrastructure used by applications (816) and / or other components and / or layers. Libraries (820) provide functionality that enables other software components to perform tasks in a way that is easier than directly interfacing with the underlying operating system (802) functionality (e.g., kernel (822), services (824) and / or drivers (826)). Libraries (820) may include system libraries (844) (e.g., C standard library) that can provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the libraries (820) may include API libraries (846) such as media libraries (e.g., libraries for supporting the presentation and manipulation of various media formats such as MPREG4, H.264, MP3, AAC, AMR, JPG, PNG), graphics libraries (e.g., OpenGL frameworks that can be used to render 2D and 3D graphic content on a display), database libraries (e.g., SQLite that can provide various relational database functions), web libraries (e.g., WebKit that can provide web browsing functionality). The libraries (820) may also include a wide variety of other libraries (848) that provide many different APIs to applications (816) and other software components / modules.

[0060] Frameworks / middleware (818) (sometimes referred to as middleware) provide a higher-level common infrastructure that can be used by applications (816) and / or other software components / modules. For example, frameworks / middleware (818) can provide various graphical user interface (GUI) functions, high-level resource management, high-level location services, etc. Frameworks / middleware (818) can provide a wide spectrum of other APIs that can be used by applications (816) and / or other software components / modules, some of which may be specific to a specific operating system (802) or platform.

[0061] Applications (816) include built-in applications (838) and / or third-party applications (840). Examples of representative built-in applications (838) may include, but are not limited to, contact applications, browser applications, book reader applications, location applications, media applications, messaging applications, and / or game applications. Third-party applications (840) may include applications developed using an ANDROID™ or IOS™ software development kit (SDK) by an entity other than the vendor of a specific platform, and may be mobile software running on a mobile operating system such as IOS™, ANDROID™, WINDOWS® Phone, or other mobile operating systems. Third-party applications (840) may initiate API calls (808) provided by a mobile operating system (e.g., operating system (802)) to facilitate the functionality described herein.

[0062] Applications (816) may use built-in operating system functions (e.g., kernel (822), services (824) and / or drivers (826)), libraries (820), and frameworks / middleware (818) to create user interfaces for interacting with users of the system. Alternatively or additionally, in some systems, interaction with the user may occur through a presentation layer such as a presentation layer (814). In these systems, application / component "logic" may be separated from the modes of application / component that interact with the user.

[0063] FIG. 9 is a block diagram illustrating components of a machine (900) according to some exemplary embodiments capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and performing one or more of the methodologies discussed herein. Specifically, FIG. 9 represents a schematic representation of a machine (900) in an exemplary form of a computer system, in which instructions (910) (e.g., software, program, application, applet, app, or other executable code) may be executed to cause the machine (900) to perform one or more of the methodologies discussed herein. Accordingly, instructions (910) may be used to implement the modules or components described herein. Instructions (910) convert a general unprogrammed machine (900) into a specific machine (900) programmed to perform the described and illustrated functions in the described manner. In alternative embodiments, the machine (900) may operate as a standalone device or be coupled to other machines (e.g., networked). In a networked deployment, the machine (900) may operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment.The machine (900) may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smartphone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of executing commands (910) that specify actions to be taken by the machine (900) sequentially or otherwise. Additionally, although only a single machine (900) is exemplified, the term “machine” should also be considered to include a collection of machines that execute commands (910) individually or jointly to perform any one or more of the methodologies discussed herein.

[0064] The machine (900) may include processors (904), memory / storage (906), and I / O components (918), which may be configured to communicate with each other, for example, via a bus (902). The memory / storage (906) may include memory (914), such as main memory or other memory storage, and a storage unit (916), both of which are accessible to the processors (904), for example, via the bus (902). The storage unit (916) and the memory (914) store instructions (910) that implement one or more of the methodologies or functions described herein. The instructions (910) may also exist, wholly or partially, during the execution by the machine (900), in the memory (914), in the storage unit (916), in at least one of the processors (904) (e.g., in the processor's cache memory), or any suitable combination thereof. Accordingly, the memory (914), the storage unit (916), and the memory of the processors (904) are examples of machine-readable media.

[0065] The I / O components (918) may include a wide variety of components that provide a user interface for receiving inputs, providing outputs, generating outputs, transmitting information, exchanging information, capturing measurements, etc. The specific I / O components (918) included in a specific machine (900) will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, whereas a headless server machine may not include such a touch input device. It will be recognized that the I / O components (918) may include many other components not shown in FIG. 9. The I / O components (918) are grouped by functionality merely to simplify the discussion below, and this grouping is by no means limiting. In various exemplary embodiments, the I / O components (918) may include output components (926) and input components (928). The output components (926) may include visual components (e.g., a display such as a plasma display panel (PDP), a light-emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), acoustic components (e.g., a speaker), haptic components (e.g., a vibration motor, a resistance mechanism), other signal generators, etc.Input components (928) may include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, a photo-optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or other pointing mechanism), haptic input components (e.g., a physical button, a touch screen providing the location and / or force of a touch or touch gesture, or other haptic input components), audio input components (e.g., a microphone), etc.

[0066] In additional exemplary embodiments, I / O components (918) may include biometric components (930), motion components (934), environment components (936), or position components (938), among a wide range of other components. For example, biometric components (930) may include components that detect expressions (e.g., hand expressions, facial expressions, voice expressions, body gestures, or eye tracking), measure biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brainwaves), and identify a person (e.g., voice identification, retinal identification, face identification, fingerprint identification, or brainwave-based identification). Motion components (934) may include acceleration sensor components (e.g., accelerometers), gravity sensor components, rotation sensor components (e.g., gyroscopes), etc. Environmental components (936) may include, for example, light sensor components (e.g., photometers), temperature sensor components (e.g., one or more thermometers for detecting ambient temperature), humidity sensor components, pressure sensor components (e.g., barometers), acoustic sensor components (e.g., one or more microphones for detecting background noise), proximity sensor components (e.g., infrared sensors for detecting nearby objects), gas sensors (e.g., gas detection sensors for detecting concentrations of hazardous gases for safety or measuring pollutants in the atmosphere), or other components capable of providing indications, measurements, or signals corresponding to the surrounding physical environment. Position components (938) may include position sensor components (e.g., Global Position System (GPS) receiver components), altitude sensor components (e.g., altimeters or barometers for detecting atmospheric pressure from which altitude can be derived), direction sensor components (e.g., magnetometers), etc.

[0067] Communication can be implemented using a wide variety of technologies. I / O components (918) may include communication components (940) operable to connect the machine (900) to a network (932) or devices (920) respectively through coupling (922) and coupling (924). For example, the communication component (940) may include a network interface component, or other suitable device for interfacing with the network (932). In additional examples, the communication components (940) may include a wired communication component, a wireless communication component, a cellular communication component, a near-field communication (NFC) component, a Bluetooth® component (e.g., Bluetooth® Low Energy), a Wi-Fi® component, and other communication components that provide communication through other aspects. The devices (920) may be any of the other machine or a wide variety of peripheral devices (e.g., peripheral devices coupled via a Universal Serial Bus (USB)).

[0068] Furthermore, the communication components (940) may include components capable of detecting identifiers or operable to detect identifiers. For example, the communication components (940) may include Radio Frequency Identification (RFID) tag reader components, NFC smart tag detection components, optical reader components (e.g., optical sensors for detecting 1-dimensional bar codes such as Universal Product Code (UPC) bar codes, multi-dimensional bar codes such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D bar codes, and other optical codes), or acoustic detection components (e.g., microphones for identifying tagged audio signals). In addition, various information such as location via Internet Protocol (IP) geolocation, location via Wi-Fi® signal triangulation, and location via NFC beacon signal detection that can indicate a specific location can be derived through the communication components (940).

[0069] Glossary

[0070] In this context, "carrier signal" refers to any intangible medium capable of storing, encoding, or carrying instructions for execution by a machine, and includes digital or analog communication signals or other intangible media to facilitate the communication of such instructions. Instructions may be transmitted or received over a network using a transmission medium through a network interface device and using any one of a number of well-known transmission protocols.

[0071] In this context, "client device" refers to any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, desktop computer, laptop, portable information terminal (PDA), smartphone, tablet, ultrabook, netbook, laptop, multi-processor system, microprocessor-based or programmable consumer electronics, game console, set-top box, or any other communication device that a user can use to access a network.

[0072] In this context, “COMMUNICATIONS NETWORK” refers to one or more parts of a network that may be an ad-hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), the Internet, part of the Internet, part of the Public Switched Telephone Network (PSTN), plain old telephone service (POTS) network, cellular telephone network, wireless network, Wi-Fi® network, other types of networks, or a combination of two or more such networks. For example, a network or part of a network may include a wireless or cellular network, and a combination may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile communications (GSM) connection, or other types of cellular or wireless combination.In this example, the combination can implement any various types of data transmission technology, such as 1xRTT (Single Carrier Radio Transmission Technology), EVDO (Evolution-Data Optimized) technology, GPRS (General Packet Radio Service) technology, EDGE (Enhanced Data rates for GSM Evolution) technology, 3GPP (third Generation Partnership Project) including 3G, 4G (fourth generation wireless) networks, UMTS (Universal Mobile Telecommunications System), HSPA (High Speed ​​Packet Access), WiMAX (Worldwide Interoperability for Microwave Access), LTE (Long Term Evolution) standards, other things defined by various standards-setting organizations, other long-range protocols, or other data transmission technologies.

[0073] In this context, an "emphymeral message" refers to a message accessible for a limited duration. An emphymeral message can be text, images, videos, etc. The access time for an emphymeral message can be set by the message sender. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting method, the message is temporary.

[0074] In this context, "Machine-Readable Medium" refers to a component, device, or other type of medium capable of storing instructions and data temporarily or permanently, and may include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage (e.g., erasable and programmable read-only memory (EEPROM)) and / or any suitable combination thereof. The term "Machine-Readable Medium" should be construed to include a single medium or multiple media capable of storing instructions (e.g., a centralized or distributed database, or an associated cache and server). The term "Machine-Readable Medium" should also be construed to include any medium or combination of multiple media capable of storing instructions (e.g., code) for execution by a machine, so that when the instructions are executed by one or more processors of the machine, the machine may perform one or more of the methodologies described herein. Therefore, "machine-readable medium" refers not only to a single storage device or unit, but also to "cloud-based" storage systems or storage networks comprising multiple storage devices or units. The term "machine-readable medium" excludes the signal itself.

[0075] In this context, "component" refers to a device, physical entity, or logic having boundaries defined by function or subroutine calls, branch points, application program interfaces (APIs), or other technologies that provide the division or modularization of specific processing or control functions. Components can be combined with other components through their interfaces to execute machine processes. A component may be a packaged functional hardware unit designed to be used with other components and part of a program that performs a specific function among related functions. Components may constitute either software components (e.g., code implemented on machine-readable media) or hardware components. A "hardware component" is a tangible unit capable of performing specific actions and may be configured or arranged in a specific physical manner. In various exemplary embodiments, one or more computer systems (e.g., standalone computer systems, client computer systems, or server computer systems) or one or more hardware components of a computer system (e.g., processors or groups of processors) may be configured by software (e.g., applications or parts of applications) as hardware components that operate to perform specific operations as described herein. Hardware components may also be implemented mechanically, electronically, or any suitable combination thereof. For example, hardware components may include dedicated circuits or logic permanently configured to perform specific operations. Hardware components may be special-purpose processors such as Field-Programmable Gate Arrays (FPGAs) or Application Specific Integrated Circuits (ASICs).Hardware components may also include programmable logic or circuits that are temporarily configured by software to perform specific operations. For example, hardware components may include software executed by a general-purpose processor or another programmable processor. Once configured by such software, hardware components become specific machines (or specific components of a machine) uniquely customized to perform the configured functions and are no longer general-purpose processors. It will be recognized that the decision to implement a hardware component mechanically, in a dedicated, permanently configured circuit, or in a temporarily configured circuit (e.g., configured by software) may be driven by cost and time considerations. Accordingly, the phrase "hardware component" (or "hardware-implemented component") should be understood to encompass type entities, that is, entities that are physically configured, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a specific manner or perform the specific operations described herein. When considering embodiments in which hardware components are temporarily configured (e.g., programmed), each hardware component does not need to be configured or instantiated at any single time instance. For example, if a hardware component includes a general-purpose processor configured by software to become a special-purpose processor, the general-purpose processor may be configured as different special-purpose processors (e.g., including different hardware components) at different times. Thus, the software configures a specific processor or processors to configure a specific hardware component at one time instance and a different hardware component at a different time instance. Hardware components may provide information to other hardware components and receive information from them.Accordingly, the described hardware components may be considered to be communicably coupled. In cases where multiple hardware components exist simultaneously, communication may be achieved through the transmission of signals between or between two or more of the hardware components (e.g., via appropriate circuits and buses). In embodiments where multiple hardware components are configured or instantiated at different times, communication between such hardware components may be achieved, for example, through the storage and retrieval of information within memory structures accessible to multiple hardware components. For example, one hardware component may perform an operation and store the output of that operation in a memory device communicably coupled thereto. Subsequently, additional hardware components may access the memory device to retrieve and process the stored output. Hardware components may also initiate communication with input or output devices and manipulate resources (e.g., collections of information). Various operations of the exemplary methods described herein may be performed at least partially by one or more processors configured temporarily (e.g., by software) or permanently to perform the relevant operations. Whether configured temporarily or permanently, such processors may comprise processor-implemented components that operate to perform one or more operations or functions described herein. As used herein, "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the methods described herein may be at least partially processor-implemented, and specific processors or processors are examples of hardware. For example, at least some of the operations of the method may be performed by one or more processors or processor-implemented components.Furthermore, one or more processors may also operate to support the execution of related operations in a "cloud computing" environment or as "software as a service" (SaaS). For example, at least some of the operations may be performed by a group of computers (as examples of machines containing processors), and these operations may be accessible via a network (e.g., the Internet) and through one or more appropriate interfaces (e.g., an Application Programming Interface (API)). The execution of specific operations may not only reside within a single machine but may also be distributed among processors deployed across multiple machines. In some exemplary embodiments, the processors or components implemented by the processors may be located in a single geographic location (e.g., a home environment, an office environment, or within a server farm). In other exemplary embodiments, the processors or components implemented by the processors may be distributed across multiple geographic locations.

[0076] In this context, "processor" refers to any circuit or virtual circuit (a physical circuit emulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "instructions," "op codes," "machine code," etc.) and generates corresponding output signals applied to operate the machine. The processor may be, for example, a CPU (Central Processing Unit), a RISC (Reduced Instruction Set Computing) processor, a CISC (Complex Instruction Set Computing) processor, a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an RFIC (Radio-Frequency Integrated Circuit), or any combination thereof. The processor may also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") capable of executing instructions simultaneously.

[0077] In this context, "timestamp" refers to a sequence of characters or encoded information that identifies when a particular event occurred, providing a date and time, for example, sometimes accurate to a fraction of a second.

[0078] In this context, "LIFT" is a measure of the performance of a targeted model in predicting or classifying cases, defined as having an improved response (with respect to the population as a whole) measured for a random selection targeting model.

[0079] In this context, "phoneme alignment" is a unit of speech that distinguishes one word from another. A phoneme can consist of a sequence of closure, plosive, and aspiration events; or, a diphthong can transition from a back vowel to a front vowel. Thus, a speech signal can be described not only by which phonemes it contains, but also by the positions of the phonemes. Therefore, phoneme alignment can be described as the "time-alignment" of phonemes within a waveform to determine the appropriate sequence and position of each phoneme within the speech signal.

[0080] In this context, "audio-to-visual conversion" refers to converting audible speech signals into visible speech, where the visible speech may include a mouth shape representing the audible speech signal.

[0081] In this context, a "Time Delayed Neural Network (TDNN)" is an artificial neural network architecture whose primary purpose is to operate on sequential data. An example would be converting continuous audio into a stream of classified phoneme labels for speech recognition.

[0082] In this context, "Bidirectional Long-Short Term Memory (BLSTM)" refers to a recurrent neural network (RNN) architecture that remembers values ​​over arbitrary intervals. The stored values ​​are not modified as learning progresses. RNNs allow for forward and backward connections between neurons. BLSTM is suitable for the classification, processing, and prediction of time series when given time lags of unknown magnitude and duration between events.

Claims

Claim 1 A method comprising: receiving a bid value associated with a media overlay and a geolocation from each of a plurality of merchant computing devices; selecting a media overlay from among the media overlays based on the bid values ​​received from the plurality of merchant computing devices, wherein the selected media overlay corresponds to the highest bid value among the bid values; associating the selected media overlay with the geolocation for a predetermined period; and causing a display of the selected media overlay on the client device in response to the client device determining that it is associated with the geolocation. Claim 2 The method of claim 1, wherein the receiving step comprises: accessing a bid interface provided by a messaging server system; and receiving the bid value and the geolocation from the bid interface. Claim 3 A method according to claim 1, wherein the selecting step comprises: comparing the bid values ​​received from each merchant computing device; and identifying the media overlay corresponding to the highest bid value. Claim 4 The method according to claim 1, wherein the selecting step comprises: updating the selected media overlay in response to receiving a new highest bid value before the expiration of the predetermined period. Claim 5 The method of claim 1, wherein the associating step comprises: storing a mapping between the selected media overlay, the geolocation, and the predetermined period in a database. Claim 6 A method according to claim 1, wherein determining that the client device is associated with the geolocation comprises: accessing geolocation data from the client device; comparing the geolocation data with the associated geolocation; and determining a match between the geolocation data and the associated geolocation. Claim 7 The method according to claim 1, wherein the predetermined period comprises: a duration specified by a merchant associated with the media overlay; or a default duration set by a messaging server system. Claim 8 The method of claim 1 further comprises the steps of: receiving a user input selecting the displayed media overlay from the client device; and transmitting a message including the selected media overlay to one or more recipients. Claim 9 In claim 1, the method comprises: audio and visual content; or visual effects. Claim 10 A system comprising: memory; and at least one hardware processor coupled to said memory and comprising instructions, wherein said instructions cause said system to perform operations including: receiving a bid value associated with a media overlay and a geolocation from each of a plurality of merchant computing devices; selecting said media overlay from among the media overlays based on the bid values ​​received from said merchant computing devices, wherein said selected media overlay corresponds to the highest bid value among said bid values; associating said selected media overlay with said geolocation for a predetermined period; and causing said display of said selected media overlay on said client device in response to said client device determining that said the selected media overlay is associated with said geolocation. Claim 11 In paragraph 10, the receiving operation comprises: an operation to access a bidding interface provided by the system; and an operation to receive the bid value and the geolocation from the bidding interface, the system. Claim 12 In claim 10, the selected operation comprises: an operation of comparing the bid values ​​received from each merchant computing device; and an operation of identifying the media overlay corresponding to the highest bid value, the system. Claim 13 In paragraph 10, the selected action comprises: an action of updating the selected media overlay in response to receiving a new highest bid value before the expiration of the predetermined period, the system. Claim 14 In claim 10, the above-mentioned associating operation comprises: a system that stores a mapping between the selected media overlay, the geolocation, and the predetermined period in a database. Claim 15 In claim 10, the determining operation comprises: an operation of accessing geolocation data from the client device; an operation of comparing the geolocation data with the associated geolocation; and an operation of determining a match between the geolocation data and the associated geolocation, a system. Claim 16 In paragraph 10, the above-mentioned predetermined period comprises: a duration specified by a merchant; or a default duration set by the system. Claim 17 A non-transient machine-readable storage medium comprising instructions, wherein, when the instructions are executed by one or more processors of a machine, the machine performs operations including: receiving a bid value associated with a media overlay and a geolocation from each of a plurality of merchant computing devices; selecting the media overlay from among the media overlays based on the bid values ​​received from the plurality of merchant computing devices, wherein the selected media overlay corresponds to the highest bid value among the bid values; associating the selected media overlay with the geolocation for a predetermined period; and causing the display of the selected media overlay at the client device in response to the client device determining that the selected media overlay is associated with the geolocation. Claim 18 In claim 17, the receiving operation comprises: an operation to access a bid interface provided by a messaging server system; and an operation to receive the bid value and the geolocation from the bid interface, a non-transient machine-readable storage medium. Claim 19 In claim 17, the selecting operation comprises: an operation of comparing the bid values ​​received from each merchant computing device; and an operation of identifying the media overlay corresponding to the highest bid value, a non-transient machine-readable storage medium. Claim 20 A non-transient machine-readable storage medium according to claim 17, wherein the operations further include: receiving user input selecting the displayed media overlay from the client device; and transmitting a message containing the selected media overlay to one or more recipients.

Images