Contextual media filter search

Abstract

A method for receiving input onto a graphical user interface at a client device, capturing an image frame at the client device, the image frame including a depiction of an object, identifying the object within the image frame, accessing media content associated with the object within a media repository in response to identifying the object, and causing presentation of the media content within the image frame at the client device.

Description

[0001] Priority requirements

[0002] This application claims priority to U.S. Patent Application No. 16 / 814,221, filed March 10, 2020, and U.S. Provisional Patent Application No. 62 / 826,679, filed March 29, 2019, which are incorporated herein by reference in their entirety. Technical Field

[0003] Embodiments of this disclosure generally relate to mobile computing technologies, and more specifically, but not in a limiting way, to generating and displaying dynamic contextual media content at a client device. Background Technology

[0004] Augmented reality (AR) is a real-time, direct or indirect view of the physical real-world environment, where elements of the physical real-world environment are enhanced through computer-generated sensory input. Attached Figure Description

[0005] To facilitate identification of any particular element or action being discussed, one or more of the most significant digits in the reference numerals refer to the figure number in which the element is first introduced.

[0006] In accompanying drawings that are not necessarily drawn to scale, the same numbers may describe similar parts in different figures. The same numbers with different letter suffixes may represent different instances of similar components. Some embodiments are shown in the accompanying drawings by way of example, not limitation.

[0007] To facilitate identification of any particular element or action being discussed, one or more of the most significant digits in the reference numerals refer to the figure number in which the element is first introduced.

[0008] Figure 1 This is a graphical representation of a networking environment in which the present disclosure can be deployed, according to some example embodiments.

[0009] Figure 2 This is a graphical representation of a messaging client application based on some example embodiments.

[0010] Figure 3 It is a graphical representation of the data structure maintained in the database according to some example embodiments.

[0011] Figure 4 This is a graphical representation of messages based on some example embodiments.

[0012] Figure 5 This is a flowchart of an access restriction process based on some example embodiments.

[0013] Figure 6This is a block diagram illustrating components of a context media filter system according to some example embodiments.

[0014] Figure 7 This is a flowchart depicting a method for causing the display of a context media filter according to certain example embodiments.

[0015] Figure 8 This is an illustration of objects within an image frame based on some example embodiments.

[0016] Figure 9 This is an illustration of a contextual media filter system within a user interface based on some example embodiments.

[0017] Figure 10 This is a block diagram illustrating a software architecture in which the present disclosure can be implemented according to some example embodiments.

[0018] Figure 11 It is a schematic representation of a machine in the form of a computer system according to some example embodiments, in which a set of instructions can be executed to cause the machine to perform any or more methods discussed. Detailed Implementation

[0019] For purposes of explanation, specific configurations and details have been set forth to provide a thorough understanding of the subject matter. However, it will be apparent to those skilled in the art that embodiments of the described subject matter may be practiced without the specific details presented herein, or in various combinations thereof. Furthermore, well-known features may be omitted or simplified to avoid obscuring the described embodiments. Various examples may be given throughout the description. These are merely descriptions of specific embodiments. The scope or meaning of the claims is not limited to the examples given.

[0020] AR systems provide users with a real-time, direct or indirect view of the physical, real-world environment within a graphical user interface (GUI), where elements of the view are enhanced through computer-generated sensory input. For example, an AR interface can present media content at a location within the display of a view of the real-world environment, making the media content appear to interact with elements in the real-world environment.

[0021] This disclosure describes a contextual media filter system configured to generate and display contextual media filters at a client device. For example, the contextual media filter system is configured to perform operations including: receiving input from a graphical user interface at the client device; acquiring an image frame (the image frame including a depiction of an object) at the client device; identifying the object within the image frame; accessing media content associated with the object in a media repository in response to the identified object; and causing the media content within the image frame to be rendered at the client device.

[0022] Media content can include animated graphics exchange format (GIF) images of various shapes, sizes, and themes. In some embodiments, the contextual media filtering system can communicate with a media repository that includes a collection of categorized and tagged media content, wherein media content within the collection is tagged or labeled based on its attributes. For example, media content can be tagged with labels that identify object categories (e.g., "food" or "basketball"), such that a reference to an object category corresponds to a set of media content within the collection.

[0023] In some embodiments, in response to receiving input selecting a user option from a set of user options, the contextual media filtering system acquires an image from a client device, wherein the image frame includes a depiction of one or more objects in a real-world environment. For example, the contextual media filtering system may detect one or more local features (e.g., a first set of features) within the image frame, wherein the local features identify objects associated with the objects depicted in the image frame, or in further embodiments, one or more image and text recognition techniques may be employed to identify the objects depicted in the image frame. Based on object recognition, the contextual media filtering system obtains a set of representative features associated with the local features to compute global features of the image frame (e.g., a second set of features). The contextual media filtering system can use the global features and query a media repository to identify media content based on the local features and representative features. The contextual media filtering system further uses the local features to perform geometric verification of the media content and eliminates spurious media content from the media content.

[0024] In response to identifying a set of media content from a media repository, the contextual media filtering system generates and causes that set of media content to be displayed at the client device. In some embodiments, to generate the rendering of the set of media content, the contextual media filtering system obtains a media template that provides a rendering format and layout for the set of media content. For example, the media template may define the positions within an image frame at the client device where the set of media content is rendered. In some embodiments, the rendering of the set of media content within an image frame at the client device may be based on the positions (or multiple positions) of objects identified within the image frame.

[0025] Figure 1 This is a block diagram illustrating an exemplary messaging system 100 for exchanging data (e.g., messages and associated content) over a network. Messaging system 100 includes instances of multiple client devices 102, each client device 102 hosting multiple applications including a messaging client application 104. Each messaging client application 104 is communicatively coupled to other instances of messaging client application 104 and a messaging server system 108 via a network 106 (e.g., the Internet).

[0026] Messaging client application 104 can communicate and exchange data with another messaging client application 104 and with messaging server system 108 via network 106. The data exchanged between messaging client applications 104 and between messaging client application 104 and messaging server system 108 includes functions (e.g., commands that call functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0027] Messaging server system 108 provides server-side functionality to specific messaging client application 104 via network 106. Although some functions of messaging system 100 are described herein as being performed by messaging client application 104 or by messaging server system 108, the location of some functions within messaging client application 104 or messaging server system 108 is a design choice. For example, it is technically preferred to first deploy certain technologies and functions within messaging server system 108, but subsequently migrate those technologies and functions to messaging client application 104 in which client device 102 has sufficient processing power.

[0028] The messaging server system 108 supports various services and operations provided to the messaging client application 104. These operations include sending data to and receiving data from the messaging client application 104, and processing data generated by the messaging client application 104. For example, this data may include message content, client device information, geolocation information, media comments and overlays, message content persistence conditions, social network information, and live event information. Data exchange within the messaging system 100 is invoked and controlled via functions available through the user interface (UI) of the messaging client application 104.

[0029] Now, specifically, we turn to the messaging application server system 108. An application programming interface (API) server 110 is coupled to application server 112 and provides a programming interface to application server 112. Application server 112 is communicatively coupled to database server 118, which facilitates access to database 120, where data associated with the messages processed by application server 112 is stored.

[0030] Application Programming Interface (API) server 110 receives and sends message data (e.g., commands and message payloads) between client device 102 and application server 112. Specifically, API server 110 provides a set of interfaces (e.g., routines and protocols) that can be invoked or queried by messaging client application 104 to call functions of application server 112. Application Programming Interface (API) server 110 exposes various functions supported by application server 112, including account registration, login functionality, sending messages from a specific messaging client application 104 to another messaging client application 104 via application server 112, sending media files (e.g., images or videos) from messaging client application 104 to messaging server application 114, and settings for possible access by another messaging client application 104, media data collections (e.g., stories), retrieval of the friend list of the user of client device 102, retrieval of the collection, retrieval of messages and content, adding and deleting friends in the social graph, locating friends within the social graph, and opening application events (e.g., application events related to messaging client application 104).

[0031] Application server 112 hosts many applications and subsystems, including messaging server application 114, image processing system 116, social networking system 122, and contextual media filtering system 124.

[0032] According to some exemplary embodiments, the contextual media filtering system 124 is configured to acquire image frames including depictions of objects from a client device 102, identify one or more objects within the image frame based on the depictions of the objects, identify one or more image features based on the objects, obtain a set of media content based on the image features, and cause the presentation of the set of media content within the image frame at the client device 102. Further details of the contextual media filtering system 124 can be found below. Figure 6 Found it.

[0033] Messaging server application 114 implements various messaging techniques and functions, particularly those related to the aggregation and other processing of content (e.g., text and multimedia content) contained in messages received from multiple instances of messaging client application 104. As will be described in further detail, text and media content from multiple sources can be aggregated into content collections (e.g., referred to as stories or galleries). Messaging server application 114 can then provide these collections to messaging client application 104. Given the hardware requirements for this processing, messaging server application 114 may also perform additional processor- and memory-intensive data processing on the server side.

[0034] Application server 112 also includes an image processing system 116, which is dedicated to performing various image processing operations, typically related to images or videos received within the payload of messages from message passing server application 114.

[0035] Social network system 122 supports various social network functions and services, and makes these functions and services available to messaging server application 114. To this end, social network system 122 maintains and accesses entity graph 304 (such as...) within database 120. Figure 3 (As shown). Examples of functions and services supported by the social networking system 122 include identifiers of other users in the messaging system 100 who are related to or "follow" a particular user, as well as identifiers of other entities and interests of the particular user.

[0036] Figure 2 This is a block diagram illustrating further details of a messaging system 100 according to an example embodiment. Specifically, the messaging system 100 is shown to include a messaging client application 204 and an application server 212, which in turn embody several subsystems, namely a short-timer system 202, a collection management system 206, and an annotation system 208.

[0037] The short-timer system 202 is responsible for performing temporary access to content permitted by the messaging client application 204 and the messaging server application 114. To this end, the short-timer system 202 incorporates a number of timers that, based on durations and display parameters associated with messages or sets of messages (e.g., stories), selectively display messages and related content and enable access to messages and related content via the messaging client application 204. Further details regarding the operation of the short-timer system 202 are provided below.

[0038] The collection management system 206 is responsible for managing media collections (e.g., text, images, video, and audio data). In some examples, collections of content (e.g., messages including images, videos, text, and audio) can be organized into "event galleries" or "event stories." Such collections can be available for a specified time period, such as the duration of an event related to the content. For example, content related to a concert can be provided as a "story" during the duration of the concert. The collection management system 206 can also be responsible for publishing icons that provide notification of the existence of a specific collection to the user interface of the messaging client application 204.

[0039] The collection management system 206 further includes a curation interface 210, which allows collection managers to manage and curate specific collections of content. For example, the curation interface 210 enables event organizers to curate collections of content related to a specific event (e.g., removing inappropriate content or redundant messages). Additionally, the collection management system 206 employs machine vision (or image recognition technology) and content rules to automatically curate content collections. In some embodiments, users may be paid to include user-generated content in the collection. In this case, the curation interface 210 operates to automatically pay the user for using the user's content.

[0040] Annotation system 208 provides various functions that enable users to annotate or otherwise modify or edit media content associated with messages. For example, annotation system 208 provides functions related to the generation and publication of media overlays for messages processed by messaging system 100. Annotation system 208 is operable to provide media overlays or supplements (e.g., image filters) to messaging client application 204 based on the geographic location of client device 102. In another example, annotation system 208 is operable to provide media overlays to messaging client application 204 based on other information (e.g., the social network information of the user of client device 102). Media overlays may include audio and visual content as well as visual effects. Examples of audio and visual content include images, text, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects may be applied to media content items (e.g., photos) at client device 102. For example, media overlays may include text that can be overlaid on a photograph taken by client device 102. In another example, media overlays include location identifier overlays (e.g., Venice Beach), live event names, or business name overlays (e.g., beach cafe). In another example, annotation system 208 uses the geographic location of client device 102 to identify media overlays, which include the name of a merchant at the geographic location of client device 102. Media overlays may include other tags associated with the merchant. Media overlays may be stored in database 120 and accessible through database server 118.

[0041] In one example embodiment, annotation system 208 provides a user-based publishing platform that allows users to select geographic locations on a map and upload content associated with those locations. Users can also specify an environment in which specific media coverage should be provided to other users. Annotation system 208 generates media coverage that includes the uploaded content and associates it with the selected geographic location.

[0042] In another example embodiment, annotation system 208 provides a merchant-based publishing platform that enables merchants to select specific media coverage associated with geographic locations through a bidding process. For example, annotation system 208 associates the media coverage of the highest-bidder merchant with the corresponding geographic location within a predetermined time period.

[0043] Figure 3 This is a schematic diagram illustrating a data structure 300 that can be stored in a database 316 of a messaging server system 108 according to certain example embodiments. Although the contents of the database 316 are shown as including multiple tables, it will be understood that the data can also be stored in other types of data structures (e.g., as an object-oriented database).

[0044] Database 316 includes message data stored in message table 314. Entity table 302 stores entity data, including entity diagram 304. Entities (the records for which entities are maintained in entity table 302) can include individuals, company entities, organizations, objects, locations, events, etc. Regardless of type, any entity involved in the data stored by messaging server system 108 can be an identifiable entity. Each entity has a unique identifier and an entity type identifier (not shown).

[0045] Entity graph 304 also stores information about the relationships and associations between entities. As an example only, these relationships can be social, professional (e.g., working in a common company or organization), interest-based, or activity-based.

[0046] Database 316 also stores annotation data, such as filters, in annotation table 312. Filters (whose data is stored in annotation table 312) are associated with and applied to videos (whose data is stored in video table 310) and / or images (whose data is stored in image table 308). In one example, a filter is an overlay displayed as an overlay on an image or video during presentation to the recipient user. Filters can be of various types, including user-selected filters from a gallery of filters presented to the sending user by messaging client application 104 when the sending user is composing a message. Other types of filters include geolocation filters (also known as geographic filters), which can be presented to the sending user based on geographic location. For example, messaging client application 104 can present a location-specific or neighborhood-specific geolocation filter within the user interface based on geographic location information determined by the GPS unit of client device 102. Another type of filter is a data filter, which can be selectively presented to the sending user by the messaging client application 104 based on other inputs or information collected by the client device 102 during message creation. Examples of data filters include the current temperature at a specific location, the current speed at which the sending user is traveling, the battery life of the client device 102, or the current time.

[0047] Other annotation data that can be stored in image table 308 is so-called shot (lens) data. A shot can be a real-time effect and sound that can be added to an image or video.

[0048] In social messaging systems, content (such as images and video clips) can be communicated as part of the messaging process. One way to customize this content is by using a shot that modifies or transforms it in some way. For example, shot data can be used to make complex additions or transformations to the content image, such as adding rabbit ears to a person's head in a video clip, adding a floating heart with a background color to a video clip, changing the proportions of a feature of a person in a video clip, or many other such transformations. To provide a unique user experience, provide attributes to the shot creator, and enable the creator to protect various aspects of the data within the messaging system, the system can provide tools for users to generate and share the shot, as well as to restrict access to the shot in various ways. As described above, a shot according to the embodiments described herein refers to modifications that can be made to a video or image. This includes real-time modifications, i.e., modifying an image while it is being captured using the device's sensors, and then displaying the modified image on the device's screen along with the shot. This also includes modifications to stored content, such as video clips in a gallery that can be modified using a shot. For example, in a creator profile with multiple shots, an authorized third-party account can use a single video clip with the help of multiple shots to see how different shots will modify the stored clip. Similarly, real-time video capture can be used in conjunction with a lens to show how the video images currently captured by the device's sensors will modify the acquired data. This data can either be displayed on the screen without being stored in memory, or the contents captured by the device's sensors can be recorded and stored in memory, regardless of whether lens modifications are made (or both).

[0049] Shot data and various systems that use and modify content using shots may therefore involve the detection of objects (e.g., faces, hands, bodies, cats, dogs, surfaces, objects, etc.), tracking these objects as they leave, enter, and move around the field of view in video frames, and modifying or transforming them while tracking. In various embodiments, different methods may be used to achieve such transformations. For example, some embodiments may involve generating a 3D mesh model of one or more objects and using animated textures and transformations of the model within the video to achieve the transformation. In other embodiments, tracking points on objects can be used to place images or textures (which may be two-dimensional or three-dimensional) at the tracked locations. In further embodiments, neural network analysis of video frames can be used to place images, models, or textures within content (e.g., images or video frames). Therefore, shot data refers to the images, models, and textures used to create transformations of content, as well as the additional modeling and analysis information required to achieve such transformations through object detection, tracking, and placement.

[0050] Real-time video processing can be performed using any type of video data (e.g., video streams, video files, etc.) stored in the memory of any type of computerized system. For example, a user can load a video file and store it in the device's memory, or the device's sensors can generate a video stream. Furthermore, the lens can be used to process any object, such as human faces and body parts, animals, or inanimate objects such as chairs, cars, or other objects.

[0051] In some embodiments, when a shot and the content to be transformed are selected, a computing device identifies the elements to be transformed by the shot and detects and tracks these elements if they exist in the video frames. Elements of the object are modified according to a modification request, thereby transforming the frames of the video stream. The transformation of the video stream frames can be performed using different methods for different kinds of transformations. For example, for a transformation of frames primarily involving changes in the form of object elements, characteristic points of each element of the object are calculated (e.g., using an Active Shape Model (ASM) or other known methods). A characteristic point-based mesh is then generated for each element of at least one element of the object. This mesh is used for subsequent stages of tracking the elements of the object in the video stream. During tracking, the aforementioned mesh for each element is aligned with the position of each element. Additional points are then generated on the mesh. A first set of first points is generated for each element based on the modification request, and a second set of points is generated for each element based on this set of first points and the modification request. The frames of the video stream can then be transformed by modifying the elements of the object based on this set of first and second points and the mesh. In this method, the background of the modified object can also be changed or deformed by tracking and modifying the background.

[0052] In one or more embodiments, a transformation that alters certain regions of an object using its elements can be performed by calculating characteristic points for each element of the object and generating a mesh based on the calculated characteristic points. Points are generated on the mesh, and then various regions based on these points are generated. The elements of the object are then tracked by aligning the regions of each element with the positions of at least one element, and the properties of the regions can be modified according to a modification request, thereby transforming frames of the video stream. Depending on the specific modification request, the properties of the aforementioned regions can be transformed in different ways. This modification may include: changing the color of the region, removing at least a portion of the region from the frames of the video stream, including one or more new objects in the region based on the modification request, modifying or deforming elements of the object or region. In various embodiments, any combination of such modifications or other similar modifications may be used.

[0053] In some embodiments, augmented reality (AR) experiences can be provided. For example, AR experiences can provide interactive experiences of real-world environments, where objects in the real-world environment are augmented with computer-generated information, such as video and audio.

[0054] In some embodiments of lenses using face detection, a specific face detection algorithm (e.g., Viola-Jones) is used to detect faces in the image. Then, an Active Shape Modeling (ASM) algorithm is applied to the facial regions of the image to detect facial feature reference points.

[0055] In other embodiments, other methods and algorithms suitable for face detection can be used. For example, in some embodiments, landmarks representing distinguishable points present in most of the images considered are used to locate features. For example, for facial landmarks, the location of the left pupil can be used. Secondary landmarks can be used if the initial landmarks are not recognizable (e.g., if the person is wearing an eye patch). The landmark recognition process can be used for any such object. In some embodiments, a set of landmarks forms a shape. The shape can be represented as a vector using the coordinates of the points in the shape. One shape is aligned with another shape using a similarity transformation (allowing translation, scaling, and rotation) that minimizes the average Euclidean distance between the points of the shapes. The mean shape is the mean of the aligned training shapes.

[0056] In some embodiments, a search for landmarks begins from the mean shape aligned with the size and position of the face determined by the global face detector. This search then repeats the steps of proposing temporary shapes by adjusting the positions of the shape points by template matching the image texture around each point and then conforming the temporary shape to the global shape model until convergence occurs. In some systems, individual template matching is unreliable, and the shape model aggregates the results of weak template matchers to form a stronger overall classifier. The entire search is repeated at each level of the image pyramid, from coarse resolution to fine resolution.

[0057] Embodiments of the transformation system can acquire image or video streams on a client device and perform complex image processing locally on a client device such as client device 102, while maintaining a suitable user experience, computation time, and power consumption. Complex image processing may include changes in size and shape, emotion transformation (e.g., changing a face from frowning to smiling), state transformation (e.g., aging an object, reducing apparent age, changing gender), style transformation, application of graphical elements, and any other suitable image or video processing implemented by a convolutional neural network configured to execute efficiently on the client device.

[0058] In some example embodiments, the lens can be used by a system where a user can capture images or video streams (e.g., selfies) using a client device 102, which has a neural network operating as part of a messaging application 104 running on the client device 102. A transformation system running in the messaging application 104 determines the presence of a face in the image or video stream and provides a modification icon associated with the lens, or the lens can exist in a manner associated with the interface described herein. The modification icon includes changes that may be the basis for modifying the user's face in the image or video stream as part of the lens operation. After a modification icon is selected, the transformation system initiates a process for transforming the user's image to reflect the selected modification icon (e.g., generating a smile for the user). In some embodiments, after an image or video stream is captured and a specified modification is selected, the modified image or video stream can be presented in a graphical user interface displayed on a mobile client device. The transformation system can implement a sophisticated convolutional neural network on a portion of the image or video stream to generate and apply the selected modification. That is, the user can capture an image or video stream and the result of the modification can be presented to the user in real time or near real time after a modification icon is selected. Furthermore, modifications can be persistent when a video stream is captured and the selected modification icon remains active. Machine learning neural networks can be used to achieve such modifications.

[0059] In some embodiments, the graphical user interface presenting the modifications performed by the transformation system may provide the user with additional interactive options. These options may be based on the interface used to initiate content acquisition and lens selection (e.g., initiated from a content creator user interface). In various embodiments, the modifications may be persistent after the initial selection of the modification icon. The user can turn the modifications on or off by clicking or otherwise selecting a face modified by the transformation system and save it for later viewing or browsing to other areas of the imaging application. In cases where the transformation system modifies multiple faces, the user can globally turn the modifications on or off by clicking or selecting a single face modified and displayed within the graphical user interface. In some embodiments, a single face in a set of multiple faces can be modified individually, or such modifications can be toggled individually by clicking or selecting a single face or a series of single faces displayed within the graphical user interface.

[0060] As mentioned above, video table 310 stores video data, which in one embodiment is associated with messages (whose records are maintained within message table 314). Similarly, image table 308 stores image data associated with messages (whose message data is stored in entity table 302). Entity table 302 can associate various annotations from annotation table 312 with various images stored in image table 308 and various videos stored in video table 310.

[0061] Story table 306 stores data about collections of messages and related images, videos, or audio data, compiled into collections (e.g., stories or galleries). The creation of a specific collection can be initiated by a specific user (e.g., each user maintaining records in entity table 302). A user can create a "personal story" in the form of a collection of content that the user has created and sent / broadcast. For this purpose, the user interface of messaging client application 104 can include user-selectable icons to allow the sending user to add specific content to his or her personal story.

[0062] Collections can also constitute "live stories," which are collections of content created by multiple users manually, automatically, or using a combination of manual and automatic techniques. For example, a "live story" can constitute a curated stream of user-submitted content from various locations and events. For instance, options can be presented to a user (whose client device has location services enabled and is at a shared location event at a specific time) via the user interface of messaging client application 104 to contribute content to a specific live story. Live stories can be identified by messaging client application 104 based on the user's location. The end result is telling "live stories" from a public perspective.

[0063] Another type of content collection is called a "location story," which allows users whose client device 102 is located within a specific geographic location (e.g., at a university or on a university campus) to contribute to a specific collection. In some embodiments, contributing to a location story may require secondary authentication to verify that the end user belongs to a specific organization or other entity (e.g., is a student on a university campus).

[0064] Figure 4 This is a schematic diagram illustrating the structure of a message 400 generated by a messaging client application 104 according to some embodiments for communication with another messaging client application 204 or a messaging server application 114. The content of a particular message 400 is used to populate a message table 314 stored in a database 120 that is accessible by the messaging server application 114. Similarly, the content of message 400 is stored in memory as "in-transit" or "in-flight" data for the client device 102 or application server 112. Message 400 is shown to include the following components:

[0065] Message Identifier 402: A unique identifier that identifies message 400.

[0066] Message text payload 406: The text that will be generated by the user via the user interface of the client device 102 and included in message 400.

[0067] Image payload 408: Image data acquired by the camera component of the client device 102 or obtained from the memory component of the client device 102 and included in message 400.

[0068] Message video payload 412: Video data captured by the camera component or obtained from the memory component of the client device 102 and included in message 400.

[0069] Message audio payload 416: Audio data captured by the microphone or obtained from the memory component of the client device 102 and included in message 400.

[0070] Message annotation 420: Annotation data (e.g., filters, stickers or other enhancements) that represents annotations to be applied to message image payload 408, message video payload 412 or message audio payload 416 of message 400.

[0071] Message duration parameter 424: A parameter value, in seconds, indicating the amount of time that the content of the message (e.g., message image payload 408, message video payload 412, message audio payload 416) will be presented to the user or accessible to the user via the messaging client application 104.

[0072] Message geolocation parameter 426: Geographic location data (e.g., latitude and longitude coordinates) associated with the content payload of message 400. Multiple message geolocation parameter 426 values ​​may be included in the payload, each of which is associated with a corresponding content item included in the content (e.g., a specific image in message image payload 408 or a specific video in message video payload 412).

[0073] Message Story Identifier 428: An identifier value used to identify one or more content sets (e.g., "story") associated with a specific content item in the message image payload 408 of message 400. For example, multiple images within the message image payload 408 may each be associated with multiple content sets using their own identifier values.

[0074] Message Label 430: Each message 400 can be labeled with multiple labels, each label indicating the subject of the content included in the message payload. For example, if a specific image included in the message image payload 408 depicts an animal (e.g., a lion), a label value indicating the relevant animal can be included in the message label 430. Label values ​​can be manually generated based on user input or automatically generated using, for example, image recognition.

[0075] Message sender identifier 432: An identifier (e.g., a messaging system identifier, email address, or device identifier) ​​that indicates the user of the client device 102 on which message 400 is generated and from which message 400 is sent.

[0076] Message recipient identifier 434: An identifier (e.g., a messaging system identifier, email address, or device identifier) ​​indicating the user of the client device 102 to which message 400 is addressed.

[0077] The content (e.g., values) of each component of message 400 can be pointers to locations in tables storing content data values. For example, image values ​​in message image payload 408 can be pointers (or their addresses) to locations within image table 414. Similarly, values ​​in message video payload 412 can point to data stored in video table 418, values ​​stored in message annotation 420 can point to data stored in annotation table 422, values ​​stored in message story identifier 428 can point to data stored in story table 410, and values ​​stored in message sender identifier 432 and message receiver identifier 434 can point to user records stored in entity table 404.

[0078] Figure 5 This is a schematic diagram illustrating an access restriction process 500, according to which access to content (e.g., short message 502 and associated multimedia data payload) or a set of content (e.g., short message set 506) can be time-restricted (e.g., short-term).

[0079] Short message 502 is shown to be associated with message duration parameter 508, the value of which determines the amount of time the messaging client application 104 displays short message 502 to the receiving user. In one embodiment, depending on the amount of time specified by the sending user using message duration parameter 508, the receiving user may view short message 502 for a maximum of 10 seconds.

[0080] The message duration parameter 508 and the message receiver identifier 518 are shown as inputs to a message timer 514, which is responsible for determining the duration of the short message 502 to be displayed to the specific receiving user identified by the message receiver identifier 518. Specifically, the short message 502 will only be displayed to the relevant receiving user for the time period determined by the value of the message duration parameter 508. The message timer 514 is shown as providing its output to a more general short timer system 504, which is responsible for the overall timing of displaying content (e.g., short message 502) to the receiving user.

[0081] exist Figure 5The diagram illustrates short messages 502 that will be included within a short message set 506 (e.g., a set of messages in a personal story or event story). The short message set 506 has an associated set duration parameter 510, the value of which determines the duration for which the short message set 506 is presented and available to users of the messaging system 100. For example, the set duration parameter 510 might be the duration of a concert, where the short message set 506 is a set of content related to that concert. Alternatively, when the short message set 506 is created, the user (owner user or curator user) can specify the value of the set duration parameter 510.

[0082] Additionally, each short message 502 within the short message set 506 has an associated set participation parameter 512, the value of which determines the duration for which the short message 502 will be accessible within the context of the short message set 506. Therefore, a particular short message set 506 may "expire" and become inaccessible within its context before the short message set 506 itself expires according to the set duration parameter 510. The set duration parameter 510, the set participation parameter 512, and the message receiver identifier 518 all provide input to a set timer 516, which operationally first determines whether a particular short message 502 in the short message set 506 will be displayed to a specific receiving user, and if so, for how long. Note that, as a result of the message receiver identifier 518, the short message set 506 also knows the identity of the specific receiving user.

[0083] Therefore, the set timer 516 operationally controls the overall lifetime of the associated short message set 506 and the individual short messages 502 included in the short message set 506. In one embodiment, each short message 502 within the short message set 506 remains visible and accessible for a period of time specified by the set duration parameter 510. In a further embodiment, a short message 502 may expire within the context of the short message set 506 based on the set participation parameter 512. Note that even within the context of the short message set 506, the message duration parameter 508 can still determine the duration for which a particular short message 502 is displayed to the receiving user. Therefore, the message duration parameter 508 determines the duration for which a particular short message 502 is displayed to the receiving user, regardless of whether the receiving user views the short message 502 within or outside the context of the short message set 506.

[0084] The short-timer system 504 can also remove a specific short message 502 from the short message set 506 based on determining that the associated story participation parameter 512 has expired. For example, when the sending user has established a set participation parameter 512 for 24 hours from the date of publication, the short-timer system 504 will remove the associated short message 502 from the short message set 506 after the specified 24 hours. The short-timer system 504 is also operable to remove the short message set 506 from the short message set 506 when the set participation parameter 512 of each short message 502 in the short message set 506 expires or when the short message set 506 itself expires according to the set duration parameter 510.

[0085] In certain use cases, the creator of a specific short message set 506 can specify an unrestricted set duration parameter 510. In this case, the expiration of the set participation parameter 512 of the last remaining short message 502 within the short message set 506 will determine when the short message set 506 itself expires. In this case, adding a new short message 502 with a new set participation parameter 512 to the short message set 506 effectively extends the lifetime of the short message set 506 to be equal to the value of the set participation parameter 512.

[0086] In response to the short-timer system 504 determining that the short-time message set 506 has expired (e.g., is no longer accessible), the short-timer system 504 communicates with the messaging system 100 (and, in particular, the messaging client application 104) such that the markers (e.g., icons) associated with the relevant short-time message set 506 are no longer displayed in the user interface of the messaging client application 104. Similarly, when the short-timer system 202 determines that the message duration parameter 508 of a particular short-time message 502 has expired, the short-timer system 504 causes the messaging client application 104 to stop displaying the markers (e.g., icons or text identifiers) associated with the short-time message 502.

[0087] Figure 6 This is a block diagram illustrating the components of a context media filter system 124, which, according to some example embodiments, is configured to acquire image frames from a client device 102 including depictions of objects, identify one or more objects in the image frames based on the depictions of objects, identify one or more image features based on the objects, obtain a set of media content based on the image features, and cause the presentation of the set of media content to be displayed within the image frames of the client device 102.

[0088] The context media filtering system 124 is shown to include a rendering system 604, an image feature calculation system 606, a search system 608, and a geometry verification system 610, all of which are configured to communicate with each other (e.g., via a bus, shared memory, or a switch). Any one or more of these systems can be implemented using one or more processors (e.g., by configuring the one or more processors to perform the functions described for the system and thus may include one or more processors).

[0089] Any one or more of the systems described can be implemented using hardware alone (e.g., one or more processors of a machine) or a combination of hardware and software. For example, any system described in Context Media Filtering System 124 may physically include an arrangement of one or more processors (e.g., a subset or subset of one or more processors of a machine) configured to perform the operations described herein for that system. As another example, any system of Context Media Filtering System 124 may include software, hardware, or both that configure an arrangement of one or more processors (e.g., in one or more processors of a machine) to perform the operations described herein for that system. Thus, different systems of Context Media Filtering System 124 may include different arrangements of the processors or a single arrangement of the processors at different points in time. Furthermore, any two or more systems of Context Media Filtering System 124 may be combined into a single system, and the functionality described herein for a single system may be subdivided into multiple systems. Moreover, systems described herein as implemented within a single machine, database, or device, according to various example embodiments, may be distributed across multiple machines, databases, or devices.

[0090] Figure 7 This is a flowchart depicting a method 700, according to certain example embodiments, that results in the display of a contextual media filter at a client device 102. The operation of method 700 can be described above regarding... Figure 6 The system described is used to execute it. For example... Figure 7 As shown, method 700 includes one or more operations 702, 704, 706, 708 and 710.

[0091] In operation 702, communication system 602 receives input from client device 102. In some embodiments, client device 102 includes a touchscreen display configured to receive tactile input. Therefore, input can include tactile input on the display of client device 102, wherein the tactile input provides indication of a point or location within a graphical user interface (GUI). For example, a user of client device 102 can provide input to identify an object or location depicted within the GUI by providing tactile input to a location along the display corresponding to the positioning of an object or location within the GUI.

[0092] In some embodiments, the input may include one or more input characteristics, including input pressure, input duration, and input position. In this embodiment, the communication system 602 may compare one or more input characteristics to a set of thresholds. For example, in response to detecting that an input characteristic exceeds a threshold, the communication system 602 may signal one or more systems of the context media filter system 124 to perform certain subsequent actions or operations.

[0093] At operation 704, presentation system 604 causes a GUI to be displayed at client device 102, wherein the GUI includes a display of real-time camera data streamed from or associated with a camera of client device 102. For example, client device 102 may include multiple cameras, such as one or more “front-facing” cameras and one or more “rear-facing” cameras, configured to acquire and stream image data. The real-time camera data generated by one or more cameras associated with client device 102 may include image data containing multiple image features.

[0094] In operation 706, the image feature calculation system 606 identifies objects within the image frame. The contextual media filtering system 124 compares media content within a media repository (e.g., a database) and selects the media content that is visually most similar to the objects within the image frame. Visual similarity comparison is a unified abstraction and representation of media content, enabling the calculation of similarity between two pieces of media content acquired under conditions such as different resolutions, angles, lighting, and other factors.

[0095] The image feature calculation system 606 identifies local features calculated from specific regions within an image frame. These local features are stored in an index structure. The index structure is a data structure that makes searching the database more efficient. Therefore, querying the index is faster than searching every row in the database.

[0096] Each local feature is further analyzed to compute the global features of the image frame. Global features are computed from the entire image frame, not from a specific region within the frame. In one example, global features are computed by quantizing the local features of the image frame. Quantization, involved in image processing, is a technique that compresses a series of values ​​into a single value. Global features are computed by mapping each local feature to a representative feature from a set of representative features. Each representative feature can be referred to as a "word." Global features are computed by counting the occurrences of each representative feature in the set of representative features.

[0097] For example, a representative set of features could be (A, B, C, D, ...). The image feature calculation system 606 may have already calculated 5 local features from the image. Two local features are mapped to "word" A, and three local features are mapped to "word" D. The global feature can be represented as "2A0B0C3D".

[0098] In operation 708, search system 608 accesses the media content associated with the identified object within a media repository. Search system 608 uses global features to search the media content within the index structure to retrieve the media content associated with the object from the media repository. In some examples, search system 608 uses the k-nearest neighbor algorithm when performing the search.

[0099] In operation 708, the contextual media filtering system 124 further verifies the media content using the geometric verification system 610. The media content returned by the search performed by the search system 608 may contain spurious media content matches that are visually dissimilar to the original object in the image frame. The geometric verification system 610 can use a similarity metric to determine whether a media content item is similar to the original object. The similarity metric may involve geometric constraints associated with the geometry of the original object. Therefore, the geometric verification system 610 uses geometric verification to identify and remove spurious media content from the media content.

[0100] In operation 710, the presentation system 604 causes media content to be presented within an image frame at the client device 102. For example, the presentation of a set of media content may include an AR display of that set of media content, making the acquired media content appear to interact in real time with elements depicted within the image frame.

[0101] In some embodiments, in order to generate the presentation of media content, the presentation system 604 obtains a media template, wherein the media template defines the presentation format and layout to be applied to a set of media content. For example, the media template may define the position and orientation of the media content to be presented within an image frame at the client device 102. The media content can be tagged with location information or time information in a media repository, such that references to the time of day, season, day of the week, month, or location for a particular day can identify a set of related media content.

[0102] Figure 8 This is a schematic diagram of object 800 within an image frame at the client device. In some examples, the object is a custom image that can be tracked within the image frame.

[0103] Figure 9This is a schematic diagram of a contextual media filter system 124 within the user interface 900 of client device 102. The contextual media filter system 124 can be initialized by initiating a scan of image frames. After the contextual media filter system 124 determines the media content associated with objects within the image frames, it can display a set of media content 902 that the user of the client device can swipe to select.

[0104] References to "an embodiment" or "an embodiment" in this specification mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the subject matter. Therefore, the phrases "in one embodiment" or "in an embodiment" appearing in different places throughout the specification do not necessarily refer to the same embodiment.

[0105] Figure 10 This is a block diagram 1000 illustrating a software architecture 1004, which can be installed on any one or more devices described herein. The software architecture 1004 is supported by hardware such as a machine 1002 including a processor 1020, memory 1026, and I / O components 1038. In this example, the software architecture 1004 can be conceptualized as a stack of layers, each providing specific functionality. The software architecture 1004 includes layers such as an operating system 1012, libraries 1010, a framework 1008, and an application 1006. Operationally, the application 1006 invokes API call 1050 through the software stack and receives message 1052 in response to API call 1050.

[0106] Operating system 1012 manages hardware resources and provides public services. Operating system 1012 includes, for example, kernel 1014, services 1016, and drivers 1022. Kernel 1014 serves as an abstraction layer between hardware and other software layers. For example, kernel 1014 provides functions such as memory management, processor management (e.g., scheduling), component management, network connectivity, and security settings. Services 1016 can provide other public services to other software layers. Driver 1022 is responsible for controlling the underlying hardware or interface connections with the underlying hardware. For example, driver 1022 may include a display driver, camera driver, etc. or low power consumption Drives, flash drives, serial communication drives (such as Universal Serial Bus (USB) drives), Drivers, audio drivers, power management drivers, etc.

[0107] Library 1010 provides low-level general-purpose infrastructure for application 1006 to utilize. Library 1010 may include system libraries 1018 (e.g., the C standard library) that provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. Furthermore, library 1010 may include API libraries 1024, such as media libraries (e.g., libraries supporting the rendering and manipulation of various media formats such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codecs, Joint Picture Experts Group (JPEG or JPG), or Portable Web Graphics (PNG)), graphics libraries (e.g., the OpenGL framework for rendering two-dimensional (2D) and three-dimensional (3D) graphics content on a display), database libraries (e.g., SQLite providing various relational database functionalities), web libraries (e.g., WebKit providing web browsing functionality), etc. Library 1010 may also include a wide variety of other libraries 1028 to provide application 1006 with many other APIs.

[0108] Framework 1008 provides a high-level common architecture that can be utilized by application 1006. For example, framework 1008 provides various graphical user interface (GUI) functions, high-level resource management, and advanced location nodes. Framework 1008 can provide a wide range of other APIs that can be utilized by application 1006, some of which may be specific to a particular operating system or platform.

[0109] In an example embodiment, application 1006 may include homepage application 1036, contact application 1030, browser application 1032, book reader application 1034, location application 1042, media application 1044, messaging application 1046, game application 1048, and other broadly categorized applications such as third-party application 1040. Application 1006 is a program that performs functions defined in a program. One or more applications 1006 can be created using various programming languages, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In a specific example, third-party application 1040 (e.g., using Android by an entity other than a platform-specific vendor) TM or iOS TM Applications developed using a Software Development Kit (SDK) can run on mobile operating systems such as iOS. TM ANDROID TM , Mobile software running on a phone or other mobile operating system. In this example, a third-party application 1040 can call API call 1050 provided by the operating system 1012 to perform the functions described herein.

[0110] Figure 11 This is a schematic representation of machine 1100, in which instructions 1108 (e.g., software, programs, applications, applets, application programs, or other executable code) can be executed to cause machine 1100 to perform any one or more of the methods discussed herein. For example, instructions 1108 can cause machine 1100 to perform any one or more methods described herein. Instructions 1108 transform a general, non-programmable machine 1100 into a specific machine 1100 programmed to perform the described and illustrated functions in the described manner. Machine 1100 can operate as a standalone device or can be coupled (e.g., network-connected) to other machines. In a networked deployment, machine 1100 can operate as a server machine or client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 1100 may include, but is not limited to, server computers, client computers, personal computers (PCs), tablet computers, laptop computers, netbooks, set-top boxes (STBs), PDAs, entertainment media systems, cellular phones, smartphones, mobile devices, wearable devices (e.g., smartwatches), smart home devices (e.g., smart appliances), other smart devices, network devices, network routers, network switches, network bridges, or any machine capable of executing instructions 1108 that continuously or otherwise specifies the actions that machine 1100 will take. Furthermore, although only a single machine 1100 is shown, the term "machine" can also be considered to include a collection of machines that individually or jointly execute instructions 1116 to perform any one or more of the methods discussed herein.

[0111] Machine 1100 may include processor 1102, memory 1104, and I / O components 1142, which may be configured to communicate with each other via bus 1144. In an example embodiment, processor 1102 (e.g., a central processing unit (CPU), a simplified instruction set computing (RISC) processor, a composite instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, processor 1106 and processor 1110 that execute instruction 1108. The term "processor" is intended to include multi-core processors that may include two or more independent processors (sometimes referred to as "cores") capable of executing instruction 1116 simultaneously. Although Figure 11Multiple processors 1102 are shown, but machine 1100 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with single cores, multiple processors with multiple cores, or any combination thereof.

[0112] Memory 1104 includes main memory 1112, static memory 1114, and storage cells 1116, which are accessible by processor 1102 via bus 1144. Main memory 1104, static memory 1114, and storage cells 1116 store instructions 1108 embodying any one or more of the methods or functions described herein. Instructions 1108 may also reside wholly or partially within main memory 1112, static memory 1114, machine-readable medium 1118 within storage cells 1116, at least one of processors 1102 (e.g., within the processor's cache memory), or any suitable combination thereof, during execution by machine 1100.

[0113] I / O component 1142 includes a wide variety of components for receiving input, providing output, generating output, sending information, exchanging information, acquiring measurements, etc. The specific I / O component 1142 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine may not include a touch input device. It is understood that I / O component 1142 may include... Figure 11 Many other components are not shown. In various example embodiments, I / O component 1142 may include output component 1128 and input component 1130. Output component 1128 may include visual components (e.g., displays, such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), auditory components (e.g., speakers), haptic components (e.g., vibration motors, reactive mechanisms), other signal generators, etc. Input component 1130 may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, photoelectric keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other indicating instruments), haptic input components (e.g., physical buttons, touchscreens providing touch gestures or the location and / or force of touch, or other haptic input components), audio input components (e.g., microphones), etc.

[0114] In another example embodiment, I / O component 1142 may include a variety of other components such as biometric component 1132, motion component 1134, environmental component 1136, or position component 1138. For example, biometric component 1132 includes components that detect expressions (e.g., hand gestures, facial expressions, voice expressions, body posture, or eye tracking), measure biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), and identify people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition). Motion component 1134 includes accelerometer components (e.g., accelerometers), gravity sensor components, rotation sensor components (e.g., gyroscopes), etc. Environmental component 1136 includes, for example, a lighting sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers for detecting ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones for detecting background noise), a proximity sensor component (e.g., an infrared sensor for detecting nearby objects), a gas sensor (e.g., a gas detection sensor for detecting the concentration of hazardous gases or measuring pollutants in the atmosphere for safety purposes), or other components that may provide indications, measurements, or signals corresponding to the surrounding physical environment. Location component 1138 includes a positioning sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that can detect the air pressure derived from which altitude), an orientation sensor component (e.g., a magnetometer), etc.

[0115] Communication can be implemented using a variety of technologies. I / O component 1142 further includes communication component 1140, operable to couple machine 1100 to network 1120 or device 1122 via coupler 1124 and coupler 1126, respectively. For example, communication component 1140 may include a network interface component or another suitable device interfaced with network 1120. In further examples, communication component 1140 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, etc. Components (e.g., low power) ), Components and other communication components that provide communication via other modes. Device 1122 can be any of another machine or a variety of peripheral devices (e.g., peripheral devices coupled via USB).

[0116] Furthermore, the communication component 1140 may detect identifiers or include components operable to detect identifiers. For example, the communication component 1140 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes, multi-dimensional barcodes such as Quick Response (QR) codes, Aztec codes, data matrices, digital graphics, MAX codes, PDF417, Supercodes, UCC RSS-2D barcodes, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying audio signals from the tag). Additionally, various information can be derived via the communication component 1140, such as deriving location via Internet Protocol (IP) geolocation, and via... Location can be derived through signal triangulation or by detecting NFC beacon signals that indicate a specific location.

[0117] Various memories (e.g., memory 1104, main memory 1112, static memory 1114, and / or the memory of processor 1102) and / or storage units 1116 may store one or more sets of instructions and data structures (e.g., software) embodying any one or more methods or functions described herein or used by any one or more methods or functions described herein. These instructions (e.g., instruction 1108) cause various operations of the disclosed embodiments to be implemented when executed by processor 1102.

[0118] Instruction 1108 can be sent or received via network 1120, using a transmission medium, via a network interface device (e.g., a network interface component included in communication component 1140), and using any of several known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instruction 1108 can be sent or received at device 1122 via a transmission medium via coupling 1126 (e.g., peer-to-peer coupling).

Claims

1. A computer-implemented method for displaying contextual media filters, comprising: One or more processors are used to receive input from the graphical user interface of the client device; In response to the input, one or more processors are used to acquire image frames at the client device, the image frames including depictions of objects; Using the one or more processors to identify the object within the image frame, the identification of the object within the image frame includes: Identify a first set of image features from a specific region within the image frame; Store the first set of image features in an index structure; Based on the first set of image features, a second set of image features is identified for the entire image frame. The identification of the second set of image features includes: Map each image feature in the first set of image features to a representative feature in a set of representative features; and Calculate the frequency of occurrence of each representative feature in the set of representative features to determine the second set of image features; and The second set of image features is used to search the index structure; In response to identifying the object within the image frame, the one or more processors access media content associated with the object within a media repository; and Using the one or more processors causes the media content to be rendered within the image frame at the client device.

2. The method according to claim 1, wherein, Receiving input on the graphical user interface of the client device further includes: The system initiates the process by receiving the input, which includes haptic input on the user interface of the client device. The tactile input is detected to have an attribute exceeding a threshold; and The image frame is acquired in response to the attribute of the tactile input exceeding the threshold.

3. The method according to claim 2, wherein, The attributes of the tactile input include one or more of the following: Input pressure; and Enter the duration.

4. The method according to claim 1, wherein, The first set of image features are local features identified within a predefined region within the image frame.

5. The method according to claim 1, wherein, The second set of image features are global features identified using the entire image frame.

6. The method according to claim 1, wherein, Accessing the media content associated with the object further includes: Geometric verification was used to verify the media content. Based on the geometric verification, fake media content in the media content is identified; and Remove the false media content from the media content.

7. The method according to claim 1, wherein, Based on similarity metrics, the media content associated with the object is visually similar to the object.

8. The method according to claim 1, wherein, The image frame is received from the messaging application on the client device.

9. The method according to claim 1, wherein, The media content includes multiple augmented reality experience options.

10. The method according to claim 9, wherein, The various augmented reality experience options are each associated with a number of special effects.

11. A contextual media filtering system, comprising: Memory; as well as At least one hardware processor coupled to the memory and including instructions to cause the system to perform operations, said operations including: Receive input from the graphical user interface on the client device; In response to the input, an image frame is acquired at the client device, the image frame including a depiction of an object; Identifying the object within the image frame, wherein identifying the object within the image frame includes: Identify a first set of image features from a specific region within the image frame; Store the first set of image features in an index structure; Based on the first set of image features, a second set of image features is identified for the entire image frame. The identification of the second set of image features includes: Map each image feature in the first set of image features to a representative feature in a set of representative features; and Calculate the frequency of occurrence of each representative feature in the set of representative features to determine the second set of image features; and The second set of image features is used to search the index structure; In response to identifying the object within the image frame, access media content associated with the object within a media repository; and This results in the media content being displayed within the image frame at the client device.

12. The system according to claim 11, wherein, Receiving input on the graphical user interface of the client device further includes: The system initiates the process by receiving the input, which includes haptic input on the user interface of the client device. The tactile input is detected to have an attribute exceeding a threshold; and The image frame is acquired in response to the attribute of the tactile input exceeding the threshold.

13. The system according to claim 12, wherein, The attributes of the tactile input include one or more of the following: Input pressure; and Enter the duration.

14. The system according to claim 11, wherein, The first set of image features are local features identified within a predefined region within the image frame.

15. The system according to claim 11, wherein, The second set of image features are global features identified using the entire image frame.

16. The system according to claim 11, wherein, Accessing the media content associated with the object further includes: Geometric verification was used to verify the media content. Based on the geometric verification, fake media content in the media content is identified; and Remove the false media content from the media content.

17. A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of the machine, cause the machine to perform operations, the operations including: Receive input from the graphical user interface on the client device; In response to the input, an image frame is acquired at the client device, the image frame including a depiction of an object; Identifying the object within the image frame, wherein identifying the object within the image frame includes: Identify a first set of image features from a specific region within the image frame; Store the first set of image features in an index structure; Based on the first set of image features, a second set of image features is identified for the entire image frame. The identification of the second set of image features includes: Map each image feature in the first set of image features to a representative feature in a set of representative features; and Calculate the frequency of occurrence of each representative feature in the set of representative features to determine the second set of image features; and The second set of image features is used to search the index structure; In response to identifying the object within the image frame, access media content associated with the object within a media repository; and This results in the media content being displayed within the image frame at the client device.

18. The non-transitory machine-readable storage medium according to claim 17, wherein, Receiving input on the graphical user interface of the client device further includes: The system initiates the process by receiving the input, which includes haptic input on the user interface of the client device. The tactile input is detected to have an attribute exceeding a threshold; and The image frame is acquired in response to the attribute of the tactile input exceeding the threshold.

Images