Method and system for augmented reality user interface control

Abstract

The present application proposes a system, device, medium and method for controlling a user interface using an object depicted in a user interface. The system and method initiate an augmented reality mode, which is configured to present augmented reality elements in a graphical user interface. A face is detected within the field of view of an image acquisition component and presented in the graphical user interface. In response to detecting the face, the system and method sequentially present a set of augmented reality elements in the graphical user interface. A subset of the augmented reality elements and the face can be depicted simultaneously. The movement of at least a portion of the face relative to a first augmented reality element is detected, the presentation of the first augmented reality element is modified, and at least one second augmented reality element is presented.

Description

[0001] priority

[0002] This application claims the benefit of priority to U.S. Patent Application Serial No. 15 / 853,264, filed on December 22, 2017, each of which is hereby claimed and is incorporated herein by reference in its entirety. Technical Field

[0003] Embodiments of the present disclosure generally relate to employing objects depicted within a user interface to control the user interface. More specifically, but not limitingly, the present disclosure provides systems and methods for capturing representations of real-world objects within a graphical user interface and depicting the real-world objects while employing the objects to control at least a portion of the graphical user interface. Background Art

[0004] Telecommunication applications and devices can provide communication between multiple users using various media such as text, images, sound recordings, and / or video recordings. For example, video conferencing allows two or more people to communicate with each other using a combination of software applications, telecommunication devices, and telecommunication networks. The telecommunication devices can also record video streams for transmission as messages in the telecommunication network. Some telecommunication applications enable touch-based control, gesture-based interaction with touch screens, and other user interactions with graphical user interfaces. In addition, some telecommunication applications allow users to control options for a communication session through screen or keyboard interaction while presenting images captured by a camera within a graphical user interface. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The various figures in the drawings depict only example embodiments of the present disclosure and should not be considered limiting of its scope.

[0006] Figure 1 is a block diagram illustrating a networking system according to some example embodiments.

[0007] Figure 2 is a diagram illustrating an augmented reality system according to some example embodiments.

[0008] Figure 3 is a flow chart illustrating an example method for controlling a user interface employing objects depicted within the user interface, according to some example embodiments.

[0009] Figure 4 is a user interface diagram depicting control of a user interface employing objects depicted within the user interface according to some example embodiments.

[0010] Figure 5 is a user interface diagram depicting control of a user interface employing objects depicted within the user interface according to some example embodiments.

[0011] Figure 6 is a user interface diagram depicting control of a user interface employing objects depicted within the user interface according to some example embodiments.

[0012] Figure 7 is a flow chart illustrating an example method for controlling a user interface employing objects depicted within the user interface, according to some example embodiments.

[0013] Figure 8 is a user interface diagram depicting control of a user interface employing objects depicted within the user interface according to some example embodiments.

[0014] Fig. 9 is a user interface diagram depicting control of a user interface employing objects depicted within the user interface according to some example embodiments.

[0015] Fig.10 is a user interface diagram depicting control of a user interface employing objects depicted within the user interface according to some example embodiments.

[0016] Fig.11 is a flow chart illustrating an example method for controlling a user interface employing objects depicted within the user interface, according to some example embodiments.

[0017] Fig.12 is a flow chart illustrating an example method for controlling a user interface employing objects depicted within the user interface, according to some example embodiments.

[0018] Fig.13 is a user interface diagram depicting an example mobile device and mobile operating system interface according to some example embodiments.

[0019] Fig.14 is a block diagram illustrating an example of a software architecture that may be installed on a machine according to some example embodiments.

[0020] Fig.15 is a block diagram presenting a diagrammatic representation of a machine in the form of a computer system within which a set of instructions may be executed for causing the machine to perform any of the methodologies discussed herein according to an example embodiment.

[0021] The headings provided herein are for convenience only and do not necessarily affect the scope or meaning of the terms used. DETAILED DESCRIPTION

[0022] The following description includes systems, methods, techniques, instruction sequences and computing machine program products that illustrate embodiments of the present disclosure. In the following description, for the purpose of explanation, many specific details are set forth in order to provide an understanding of the various embodiments of the subject matter of the present invention. However, it will be apparent to those skilled in the art that embodiments of the subject matter of the present invention may also be practiced without these specific details. Typically, known instruction instances, protocols, structures and techniques need not be shown in detail.

[0023] Some aspects of the present disclosure describe an augmented reality system. Some embodiments of the augmented reality system are configured to be able to control a device (e.g., a smart phone) or a graphical user interface presented on the device using objects depicted within a user interface. The augmented reality system can enable control of the device by manipulating functions or operations within a graphical user interface, a game, a software program, a tactile feedback system, a social messaging or social networking program and system, a browser, a communication network, or any other suitable interface. The augmented reality system can use an image acquisition device (e.g., a camera) associated with the device to capture a video stream, a set of frames, or a set of images including an object. When the video stream is captured, the object is depicted within a graphical user interface on a display component of the device, and the movement of the object within the video stream causes control or interaction of one or more devices with the graphical user interface depicting the object.

[0024] In the case where the augmented reality system is capable of controlling the game, the augmented reality system can collect a video stream of a control object such as a face or a hand. Then, the control object is rendered or displayed in a graphical user interface presented on the screen of a device (such as a smart phone). Then, the augmented reality system generates an augmented reality element, and the user interacts with or manipulates the augmented reality element to control or navigate the game environment. For example, in the case where the control object is a face, the front camera of the smart phone can collect a video stream including the user's face. The user's face can be shown in the game environment on the graphical user interface. Then, the augmented reality system can generate an object (e.g., a tomato, a pie, a spoon of baby food) flying or falling toward the face in the graphical user interface. Then, the user can move his face or move the smart phone while keeping his face in the field of view of the front camera to avoid the object (e.g., avoid a flying tomato or pie) or grab an object (e.g., grab a spoon of baby food in the user's mouth). After avoiding or grabbing the object, the object may disappear or be replaced by a remaining part (e.g., a splashing part of a tomato or pie). Then, the augmented reality system can generate a new object to continue the game.

[0025] In some game environments, users can also interact with persistent objects that are retained in the game environment with the control object. For example, in the case where the control object is the user's head or face, the user can place his head or face in the field of view of the front camera of the smart phone. The camera can collect a video stream including the user's face, and the augmented reality system can provide the user's face within the game environment. The augmented reality system then generates a persistent element (e.g., a ball). The augmented reality system can then throw the ball at the user's face. By moving their head or smart phone, when the user's face is in the field of view of the front camera, the augmented reality system will manipulate the movement of the ball so that the ball looks like it bounces on the user's forehead. The augmented reality system can generate additional user interface elements, such as scoreboards, score indicators, status indicators, celebration graphics, etc., to represent the number of consecutive bounces of the ball, scores, achievements, or any other suitable interactions.

[0026] The above is a specific example. Various embodiments of the present disclosure relate to an apparatus and instructions implemented by one or more processors of the apparatus to control a user interface using real-world objects captured in a video stream and depicted within the user interface. An augmented reality system is described that enables control of one or more functions of an apparatus and a graphical user interface by manipulating the position, angle, rotation, size, distance, or any other suitable aspect of an object depicted within the user interface.

[0027] Figure 1 1 is a network diagram depicting a network system 100 according to one embodiment, the network system 100 having a client-server architecture configured for exchanging data over a network. For example, the network system 100 may be a messaging system in which clients transmit and exchange data within the network system 100. The data may relate to various functions (e.g., sending and receiving text and media communications, determining geographic location, etc.) and aspects associated with the network system 100 and its users (e.g., transmitting communication data, receiving and sending indications of communication sessions, etc.). Although shown here as a client-server architecture, other embodiments may include other network architectures, such as a peer-to-peer or distributed network environment.

[0028] like Figure 1 As shown in , the network system 100 includes a social messaging system 130. The social messaging system 130 is generally based on a three-layer architecture, which includes an interface layer 124, an application logic layer 126, and a data layer 128. As understood by those skilled in the relevant computer and Internet related fields, Figure 1Each component or engine shown in the figure represents a set of executable software instructions and corresponding hardware (e.g., memory and processor) for executing the instructions, forming a hardware-implemented component or engine, and when executing the instructions acts as a special-purpose machine configured to perform a specific set of functions. In order to avoid obscuring the subject matter of the present invention with unnecessary details, the following description will be given. Figure 1 Various functional components and engines that are not closely related to conveying an understanding of the subject matter of the present invention are omitted. Of course, additional functional components and engines can be used with social messaging systems (such as Figure 1 In addition, the social messaging system shown in the example above is used together with the social messaging system shown in the example above to facilitate implementing additional functions not specifically described herein. Figure 1 The various functional components and engines depicted in the can reside on a single server computer or client device, or can be distributed across several server computers or client devices in various arrangements. Figure 1 1 and 2. Social messaging system 130 is depicted as a three-tier architecture, but the subject matter of the present invention is by no means limited to such an architecture.

[0029] like Figure 1 As shown in FIG. 1 , the interface layer 124 includes an interface component (e.g., a web server) 140 that receives requests from various client computing devices and servers, such as a client device 110 executing a client application 112, and a third-party server 120 executing a third-party application 122. In response to the received request, the interface component 140 transmits an appropriate response to the requesting device via the network 104. For example, the interface component 140 can receive a request, such as a hypertext transfer protocol (HTTP) request or other web-based application programming interface (API) request.

[0030] The client device 110 may execute a conventional web browser application or one that has been developed for a specific platform to include a variety of mobile computing devices and mobile-specific operating systems (e.g., iOS). TM ANDROID TM , PHONE). In addition, in some example embodiments, client device 110 forms all or part of augmented reality system 160, such that components of augmented reality system 160 configure client device 110 to perform a specific set of functions with respect to the operation of augmented reality system 160.

[0031] In an example, client device 110 executes client application 112. Client application 112 may provide functionality for presenting information to user 106 and communicating via network 104 to exchange information with social messaging system 130. Additionally, in some examples, client device 110 executes functionality of augmented reality system 160 to control a user interface using real-world objects captured in a video stream and depicted within the user interface.

[0032] Each client device 110 may include a computing device that includes at least a display and communication capabilities with the network 104 to access the social messaging system 130, other client devices, and third-party servers 120. Client devices 110 include, but are not limited to, remote devices, workstations, computers, general-purpose computers, Internet devices, handheld devices, wireless devices, portable devices, wearable computers, cellular or mobile phones, personal digital assistants (PDAs), smart phones, tablet computers, ultrabooks, netbooks, laptop computers, desktops, multiprocessor systems, microprocessor-based or programmable consumer electronics, game consoles, set-top boxes, network PCs, minicomputers, etc. Users 106 may be people, machines, or other components that interact with client devices 110. In some embodiments, users 106 interact with the social messaging system 130 via client devices 110. Users 106 may not be part of the networked system 100, but may be associated with client devices 110.

[0033] like Figure 1 As shown in FIG, data layer 128 has a database server 132 that facilitates access to an information repository or database 134. Database 134 is a storage device that stores data such as member profile data, social graph data (e.g., relationships between members of social messaging system 130), image modification preference data, accessibility data, and other user data.

[0034] Individuals can register with the social messaging system 130 to become members of the social messaging system 130. After registration, members can form social network relationships (e.g., friends, followers, or contacts) on the social messaging system 130 and interact with a wide range of applications provided by the social messaging system 130.

[0035] The application logic layer 126 includes various application logic components 150 that, in conjunction with the interface components 140, generate various user interfaces using data obtained from various data sources or data services in the data layer 128. The various application logic components 150 can be used to implement functions associated with various applications, services, and features of the social messaging system 130. For example, a social messaging application can be implemented with at least a portion of the application logic components 150. The social messaging application provides a messaging mechanism for users of the client device 110 to send and receive messages including text and media content (such as pictures and videos). The client device 110 can access and view messages from the social messaging application for a specified period of time (e.g., limited or unlimited). In an example, a message recipient can access a particular message for a predefined duration (e.g., specified by the message sender), which predefined duration begins when the particular message is first accessed. After the predefined duration has passed, the message will be deleted and the message recipient will no longer be able to access the message. Of course, other applications and services can be embodied in their respective application logic components 150.

[0036] like Figure 1 As shown in , the social messaging system 130 may include at least a portion of an augmented reality system 160 capable of controlling a user interface using objects depicting the user interface. Similarly, as described above, the client device 110 includes at least a portion of the augmented reality system 160. In other examples, the client device 110 may include the entire augmented reality system 160. In the event that the client device 110 includes a portion (or all) of the augmented reality system 160, the client device 110 may work alone or in cooperation with the social messaging system 130 to provide the functionality of the augmented reality system 160 described herein.

[0037] In some embodiments, the social messaging system 130 may be a short messaging system that allows for short-term communications in which content (e.g., a video clip or image) is deleted after a deletion trigger event such as viewing time or viewing completion. In such embodiments, the device uses the various components described herein in any case of generating, sending, receiving, or displaying aspects of short messages. For example, a device implementing the augmented reality system 160 may enable interactions (e.g., playing catch) between users of the social messaging system. The device may enable, generate, and send representations of interactions between users as part of the content generation of the short message. After the interaction session is complete, a deletion trigger may be enabled to delete the record or representation of the interaction.

[0038] exist Figure 2In various embodiments, the augmented reality system 160 can be implemented as a standalone system or in conjunction with the client device 110, and is not necessarily included in the social messaging system 130. The augmented reality system 160 is shown to include a mode component 210, a recognition component 220, an enhancement component 230, and a motion component 240. All or some of the components 210-240 communicate with each other, for example, via a network coupling, shared memory, etc. Each of the components 210-240 can be implemented as a single component, combined into other components, or further subdivided into multiple components. Other components not relevant to the example embodiment may also be included, but are not shown.

[0039] Figure 3 A flow chart illustrating an example method 300 for controlling a user interface employing an object depicting the user interface, according to some example embodiments, is depicted. The operations of the method 300 may be performed by components of the augmented reality system 160 and are described below for purposes of illustration.

[0040] In operation 310, the mode component 210 initiates an augmented reality mode of the client device 110. In some embodiments, the augmented reality mode is configured to present augmented reality elements within a graphical user interface presented on a display component of the client device 110. The augmented reality mode can be initiated in response to the selection of an icon or other selectable interface element. Upon selecting the icon, the mode component 210 initiates the augmented reality mode, which can include or include software configured to perform the operations described below. For example, the mode component 210 can initiate the augmented reality mode by opening an augmented reality program and executing one or more processor-readable instructions to cause the presentation of a graphical user interface on the display component of the client device 110. In some cases, the graphical user interface initially includes at least a portion of the field of view of the image acquisition component of the client device 110. For example, the graphical user interface can initially include a portion of the field of view of a front-facing camera of a smart phone.

[0041] In some embodiments, the graphical user interface initially includes one or more instructions for a user to interact with the client device 110. The instructions may indicate one or more of a set of game rules, an initial object window (e.g., an area designated for initial placement of an object of interest or a control object of the graphical user interface), a set of control operations, and any other suitable information. In some cases, the graphical user interface includes one or more optional interface elements. The optional interface elements may represent optional options for a game played using the augmented reality system 160, modifying objects presented within the field of view of the image acquisition component of the client device 110, modifying the field of view of the image acquisition component, or any other suitable optional options associated with the augmented reality system 160.

[0042] In operation 320, the recognition component 220 detects an object of interest within a field of view of an image acquisition component coupled to the client device 110. In some embodiments, Figure 4 As shown in , the object of interest is the face of a user operating or approaching the client device 110 and within the field of view of the image acquisition component. For example, the client device 110 can enable image acquisition using a forward-facing image acquisition component (e.g., a selfie camera) of the client device 110. The user of the client device 110 can be positioned within the field of view of the forward-facing image acquisition component. After the forward-facing image acquisition component initiates image acquisition of the field of view, the acquired image (e.g., a frame in a video stream) can be passed to the recognition component 220. The recognition component 220 can then detect the user's face 402 and display the face 402 within the graphical user interface 400 on the display component of the client device 110.

[0043] As described above, in some embodiments, the recognition component 220 receives or otherwise accesses one or more images depicting at least a portion of a face. In some embodiments, the recognition component 220 receives the one or more images as a video stream captured by an image acquisition component associated with the client device 110 and presented on a user interface in an augmented reality mode. The recognition component 220 may directly receive one or more images or video streams captured by the image acquisition component. In some cases, the recognition component 220 passes all or a portion of the one or more images or video streams (e.g., a set of images comprising a video stream) to one or more components of the augmented reality system 160 for further processing and manipulation, as described in more detail below.

[0044] In some embodiments, the recognition component 220 is configured to detect a predetermined object or object type. For example, the recognition component 220 can be configured to detect or recognize a person, face, facial features, hand, or any other object or object type that appears in the field of view of the image acquisition component. The recognition component 220 can be configured to detect an object or object type from a group of objects or object types. The recognition component 220 can also be configured to detect an object or object type based on a mode (e.g., an augmented reality mode). In this embodiment, the mode can be selected from a plurality of modes, each of which is associated with a specified object or object type.

[0045] The recognition component 220 can determine the object of interest based on size, position, prominence, shape, color, color change, a set of reference points, edge detection operations, object tracking operations, or any other suitable definable or detectable aspect of the object. For example, in some embodiments, the recognition component 220 can employ a set of facial tracking operations to determine one or more landmarks associated with a face located within the field of view of the image acquisition component. The recognition component 220 can determine that a face depicted within the field of view is an object of interest based on the detected landmarks. In some embodiments, the recognition component 220 applies a facial grid to the object of interest (e.g., a face) such that the location of one or more landmarks within one or more frames of a video stream is known. Once the location of the one or more landmarks is known within at least a portion of the frames of the video stream, the recognition component 220 can track the landmarks.

[0046] In operation 330, the augmentation component 230 sequentially presents a set of augmented reality elements within the graphical user interface. Figure 4 As shown in FIG, in response to the recognition component 220 detecting an object of interest (e.g., face 402) within the field of view of the image acquisition component, a set of augmented reality elements 404 are presented. The augmentation component 230 can present the set of augmented reality elements 404 in the graphical user interface 400 such that a subset of the augmented reality elements and the object of interest (e.g., the user's face 402) are depicted within the graphical user interface 400.

[0047] In some embodiments, augmentation component 230 presents one or more augmented reality elements at the initiation of a session, such as a gaming session in augmented reality mode. For example, an indication of a user interface selection to initiate a session in augmented reality mode may be communicated to augmentation component 230. Figure 4 As shown in , augmentation component 230 can then present first augmented reality element 406 with object of interest 402 within graphical user interface 400. First augmented reality element 406 can initially be presented in a first position. In some cases, when presented within graphical user interface 400, first augmented reality element 406 can be presented at a speed such that first augmented reality element 406 travels along a path within graphical user interface 400. Figure 4-6 As shown in , as one travels along the path, the first augmented reality element 406 may change position on the graphical user interface 400 in successively acquired and rendered frames depicted on the display component of the client device 110 .

[0048] In some embodiments, one or more augmented reality elements include an interactive environment. In this embodiment, face 402 can be presented in or on a graphically generated subject such as an animated character, an avatar, a suit (e.g., a space suit, a flight suit, a set of clothes), or any other suitable subject. One or more augmented reality elements may include a background, one or more interactive objects (e.g., trees, flowers, clouds, people, animals, etc.). A first portion of the augmented reality element (e.g., the background and the subject) can be presented simultaneously, and a second portion of the augmented reality element (e.g., trees, animals, and debris) can be presented sequentially in response to the movement of face 402.

[0049] In operation 340, motion component 240 detects motion of at least a portion of object of interest 402 relative to first augmented reality element 406 in the subset of augmented reality elements. In some embodiments, motion component 240 detects motion of the portion of object of interest 402 by tracking motion of reference features 408. For example, motion component 240 can detect motion of face 402 by detecting motion of reference features 408, where reference features 408 are one or more points defining a perimeter of face 402. In some cases, reference features 408 include one or more of a set of facial landmarks, facial features (e.g., a nose, one or more eyes, a mouth, a jawline, a hairline, facial reference points representing one or more features, combinations thereof, or any other suitable facial features), facial characteristics, or any other identifiable or distinguishable aspect of a face.

[0050] like Figure 4-6 As shown in FIG. , the motion component 240 can be compared with ( Figure 4 and Figure 5 ) video stream to detect motion. In detecting the motion of the object of interest 402, the motion component 240 may determine that in the first frame (e.g., Figure 4 ). The reference features 408 may be one or more of a set of landmarks, a set of edges, or any other distinguishing aspects of the object of interest 402. The motion component 240 may then determine or detect a first position of the reference features 408 in the second frame (e.g., Figure 5 ). In the event that the first position and the second position are different, the motion component 240 can detect the motion of the object of interest 402. In some embodiments, the motion component 240 determines the motion by detecting the average position of the object of interest 402 or the reference feature for multiple frames (e.g., consecutive or continuous frames or non-consecutive frames).

[0051] In some embodiments, where the augmented reality element includes an environment, the movement of a portion of the object of interest can cause the object of interest to be presented at a different or changing position within the graphical user interface. The object of interest can interact with a portion of the augmented reality element (e.g., a first augmented reality element) by changing the position of the object of interest, thereby avoiding, contacting, capturing, jumping over, or otherwise performing an action or a set of actions relative to at least one augmented reality element. For example, the object of interest can be presented in the augmented reality environment and depicted as a person skiing along a hillside. The movement of the object of interest (e.g., a face) can control the movement of a body or avatar of the graphical user interface including the object of interest to avoid obstacles (e.g., a portion of the augmented reality element), collect or contact a target (e.g., a portion of the augmented reality element) and proceed down the rendered ski slope. The movement of the object of interest can also be used to control any suitable movement through the augmented reality environment, such as flying, running, sliding, swimming, or any other suitable method of traveling or navigating. In some embodiments, the movement of the object of interest (such as a head) can be performed by moving the object of interest within the field of view of the image acquisition device while maintaining a static position of the image acquisition device. For example, a person holding a camera can keep the camera in a static position while moving his head. The motion may control graphical elements depicted on the camera's screen (eg, a smartphone display), motion or changes in the environment depicted on the screen, the generation of new elements, the progression of a game, or any other suitable change.

[0052] In operation 350, the augmentation component 230 modifies the presentation of the first augmented reality element and causes the presentation of at least one second augmented reality element in the subset of augmented reality elements. The augmentation component 230 may modify the first augmented reality element based on the motion of or proximity to the object of interest or reference feature. The augmentation component 230 may present the at least one second augmented reality element in response to the modification of the first augmented reality element. Figure 4-6 As shown in FIG. 1 , the augmented reality component 230 and the motion component 240 track the first augmented reality element 406 and the object of interest 402 within a frame or image comprising a video stream. When the first augmented reality element 406 approaches the object of interest 402, the object of interest 402 may move out of the perceived path or track of the first augmented reality element 406. Figure 6 As shown in , the first augmented reality element 406 can be modified by changing the amount of the first augmented reality element 406 depicted within the graphical user interface 400. For example, the first augmented reality element 406 can be depicted as passing behind the object of interest 402 or a portion of the user associated with the object of interest 402.

[0053] In some embodiments, Figure 6, upon modifying the first augmented reality element 406, the augmentation component 230 may cause the presentation of at least one second augmented reality element 410. Similar to the manner described above with respect to the first augmented reality element 406, the at least one second augmented reality element 410 may be presented as traveling along a track within the graphical user interface 400. In some embodiments, the at least one second augmented reality element 410 may have a shape, size, type, or other set of features similar to the first augmented reality element 406. The at least one second augmented reality element 410 may also include additional augmented reality elements indicating the modification of the first augmented reality element 406. For example, an augmented reality element may be generated and presented indicating that the first augmented reality element 406 missed the object of interest 402 or hit the object of interest 402. An augmented reality element may also be presented indicating a score or points, an increment of a number (e.g., the number of misses, the number of hits, the score or total score, etc.), a celebration effect, a color change, or any other suitable enhancement of the video stream.

[0054] Figure 7 A flow chart illustrating an example method 700 for controlling a user interface employing an object depicting a user interface according to some example embodiments is depicted. The operations of method 700 may be performed by components of augmented reality system 160. In some cases, as will be explained in more detail below, certain operations of method 700 may be performed using or as sub-operations of one or more operations of method 300.

[0055] In operation 710, one or more of the recognition component 220 and the motion component 240 determines a facial location of a reference feature for an object of interest. In some embodiments, the recognition component 220 identifies one or more facial landmarks on the object of interest (e.g., a face). The recognition component 220 applies a facial mesh to the face. One or more vertices of the facial mesh may correspond to one or more facial landmarks. The recognition component 220 may determine the facial location of the reference feature from one or more vertices or one or more landmarks of the facial mesh. For example, at least a portion of the one or more vertices may be associated with a specified facial feature. After applying the facial mesh to the face, the recognition component 220 may select a vertex or facial landmark for the reference feature based on a known association of the facial feature with the vertices of the facial mesh.

[0056] In some cases, recognition component 220 determines the facial location of the reference feature by identifying a plurality of facial landmarks on the subject of interest. After identifying the facial landmarks, recognition component 220 selects one or more facial landmarks associated with the reference feature. The one or more facial landmarks may define the reference feature, represent one or more points or aspects of the reference feature, or otherwise be associated with the reference feature.

[0057] For example, Figure 8 , the augmentation component 230 can initially present a graphical user interface 800 that depicts an object of interest 802 (e.g., a user's face) within the field of view of an image acquisition component 804 of the client device 110. The recognition component 220 then determines the facial location of a reference feature 806 (e.g., a mouth) depicted on the object of interest 802. The augmentation component 230 then presents a first augmented reality element 808. As will be described in more detail below, the first augmented reality element 808 can be presented at a first location and depicted as changing location from the first location to a second location along a trajectory, path, or otherwise.

[0058] In operation 720, the augmentation component 230 determines the spatial location of the first augmented reality element within the graphical user interface. Figure 8 As shown in , the augmentation component 230 can present a first augmented reality element 808 and determine or identify a spatial position of the first augmented reality element 808 in the graphical user interface 800 at a time after the first augmented reality element 808 is presented. In some embodiments, the spatial position can be determined at a time corresponding to a reference feature changing from a first state to a second state. The spatial position of the first augmented reality element 808 can be determined as a position within a grid defined for the graphical user interface, a position or pixel position corresponding to a set of pixels depicting the first augmented reality element, a set of coordinates for the first augmented reality element 808 or a point on or within the first augmented reality element 808, or any other suitable definable spatial position within the graphical user interface 800.

[0059] In some embodiments, the spatial position of the first augmented reality element is determined relative to a reference feature or a facial position of the reference feature. For example, the spatial position of the first augmented reality element may be determined as a measure of a distance to a portion of the reference feature, a number of pixels between the first augmented reality element and the reference feature, or any other suitable relative position or measurement. In some cases, augmentation component 230 may determine the spatial position of the first augmented reality element based on or in response to a motion of the first augmented reality element.

[0060] For example, Figure 8 , the first augmented reality element 808 is depicted as being in a first position. The augmentation component 230 and the motion component 240 can detect the motion of the first augmented reality element 808 by comparing the position of the face (e.g., the object of interest 802) in two or more frames of the video stream. In detecting the motion of the object of interest 802, the motion component 240 or the augmentation component 230 can determine that the reference feature 806 is located in the first frame (e.g., Figure 8), a first position in the second frame (e.g., a set of landmarks, a set of edges, or any other distinguishing aspect of the object of interest 802 or the reference feature 806). The motion component 240 can then determine or detect that the object of interest 802 or the reference feature 806 is located in the second frame (e.g., Fig. 9 ) in a second position in the image. When the first position and the second position are different, the motion component 240 can detect the motion of the object of interest 802 or the reference feature 806.

[0061] In some embodiments, the motion component 240 determines motion by detecting an average position of the object of interest 802 or the reference feature 806 for a plurality of frames (e.g., consecutive or continuous frames or non-consecutive frames). In response to detecting motion for the object of interest 802, the augmentation component 230 can detect a spatial position of the first augmented reality element 808. In some cases, the spatial position can be identified as a position, location, or distance relative to the object of interest 802 or a point on the object of interest 802.

[0062] In some embodiments where the motion component 240 detects motion of the reference feature 806, the motion component 240 identifies a first state of the reference feature 806. The first state can correspond to one or more first positions of a set of landmark points corresponding to the reference feature. For example, where the reference feature is a mouth depicted on the user's face, the motion component 240 can detect a first state of the mouth in a first frame. The first state can correspond to the position of one or more pixels, landmarks (e.g., oral commissure, vermillion border, Cupid's bow, philtrum, philtral ridge, or other point), or other discernible aspects of the mouth in the first frame. The motion component can then detect a second state of the mouth for the second frame by comparing the positions determined for the first frame with the positions of the same aspects or features in the second frame.

[0063] In operation 730, augmentation component 230 determines the proximity of the spatial location to the facial location as being within a proximity threshold. In some cases, the proximity threshold is a distance from at least a portion of an object of interest or a reference feature. The proximity threshold can be defined as a distance from the object of interest or the reference feature that places at least a portion of the first augmented reality element to overlap or intersect an edge or boundary of the reference feature or the object of interest. For example, Figure 8-10 As shown in , when the first augmented reality element moves from the first position ( Figure 8 ) moves to a subsequent position (shown in ) that overlaps a portion of reference feature 806 Fig.10 ), it can be determined that the first augmented reality element 808 is within the proximity threshold.

[0064] After identifying the first state, the motion component 240 determines a change in the reference feature 806, as described above with respect to operation 720. The change in the reference feature 806 can be a change from the first state to a second state that corresponds to one or more second positions of at least one landmark point in the set of landmark points. As described above, the second position of the at least one landmark point can be a position of the at least one landmark point in a second frame of the video stream. The motion component 240 can use any suitable motion tracking, face tracking, or object tracking operation to detect motion of the landmarks of the object of interest 802, the reference feature 806, or one or both of the object of interest and the reference feature.

[0065] For example, when the object of interest 802 is a face and the reference feature 806 is a mouth (eg, Figure 8-10 ), the motion component 240 may determine the first state of the mouth as an open state, such that the mouth is open to an extent detectable by the motion component 240, the recognition component 220, or the enhancement component 230. In some cases, the open state may be determined as an open mouth exceeding a predetermined threshold. The opening of the mouth may be determined based on the amount of black or dark pixels present within a selected area of ​​the graphical user interface. The selected area is an area defined by one or more of the mouth or a bounding box containing a facial area associated with the mouth. The opening of the mouth may also be determined based on the distance between or near one or more facial landmark points of the mouth. The second state may be a closed state of the mouth. In this embodiment, the change of the reference feature 806 (e.g., the mouth) from the first state to the second state may be a representation of the mouth moving from an open position to a closed position or toward a closed position.

[0066] In some embodiments, the proximity of the spatial location is determined in response to the motion of the object of interest 802. The motion component 240 can detect the motion by continuously tracking the object of interest 802, the reference feature 806, or a landmark of one or both of the object of interest 802 and the reference feature 806. The motion component 240 can also detect the motion by periodically determining the location of one or more of the object of interest 802, the reference feature 806, or one or more landmarks. When periodically determining the location, the motion component 240 can select a set of frames (e.g., reference frames) from which the location is determined. The set of reference frames can be selected based on a predetermined frame interval, the quality of the frame (e.g., focus quality, the presence of one or more specified landmarks, or any other quality metric), a frame average (e.g., one or more averaged aspects, features, or quality of a subset of frames), or any other suitable selection metric. After selecting or determining the reference frame, the motion component 240 can determine the location of the object of interest 802, the reference feature, a portion thereof, or at least one landmark within the reference frame, and compare the detected location with one or more previously detected locations in a previous reference frame. After determining the motion of the object of interest 802 , the augmentation component 230 can determine the spatial position of the first augmented reality element 808 relative to the facial position of the object of interest 802 .

[0067] In operation 740, in response to detecting the motion of the reference feature 806 changing from the first state to the second state, the augmenting component 230 modifies the presentation of the first augmented reality element 808. In some cases, as described above, the augmenting component 230 modifies the presentation of the first augmented reality element 808 based on the proximity of the spatial position to the facial position. The augmenting component 230 can modify the presentation of the first augmented reality element 808 by performing one or more operations or sub-operations upon detecting a change in state of the reference feature 806. For example, the augmenting component 230 can iteratively or gradually cover or remove a portion of the first augmented reality element 808, change one or more colors of the first augmented reality element 808, change the size of the first augmented reality element 808, or any other suitable modification.

[0068] As described above, in some embodiments, the augmentation component 230 determines at the spatial location that the first augmented reality element 806 covers at least a portion of the reference feature 806 (e.g., a mouth). The portion of the reference feature 806 covered by the first augmented reality element 808 can be located within the interior area of ​​the reference feature 806. For example, in the case where the reference feature 806 is a mouth, the interior area of ​​the mouth can be a dark area that represents an open mouth in the graphical user interface and is bounded by pixels representing lips or other parts of the mouth. Thus, in some cases, the interior area can be understood as the interior surface of the reference feature 806.

[0069] In response to the first augmented reality element 808 covering a portion of the reference feature 806, the augmentation component 230 can modify the presentation of the first augmented reality element 808 for a period of time before terminating the presentation of the first augmented reality element 808. For example, where the reference feature 806 is a mouth and the first augmented reality element 808 is a piece of candy covering a portion of the opening of the mouth, the augmentation component 230 can gradually modify the presentation of the first augmented reality element 808 (e.g., gradually changing one or more colors, gradually reducing the size, or other progressive or iterative presentation modifications) to depict the candy falling into the open mouth. After the period of time has passed, the position of the first augmented reality element 808 has changed, or any other suitable trigger, the augmentation component 230 can terminate the presentation of the first augmented reality element 808.

[0070] For example, Fig.10 As shown in FIG. 1 , the augmentation component 230 can modify the presentation of the first augmented reality element 808 relative to the object of interest 802 or a portion thereof in one or more frames of the video stream. Figure 8-10 In FIG. 800 , a first augmented reality element 806 is depicted as traveling along a trajectory across a set of frames. An object of interest 802 is detected as moving relative to one or more of the first augmented reality element 806 and a point within the graphical user interface 800 . Fig.10 , the augmentation component 230 modifies the presentation of the first augmented reality element 806 in response to the object of interest 802 acting on the first augmented reality element 806. As shown, one or more components of the augmented reality system 160 can detect the proximity of a portion of the object of interest 802 (e.g., a mouth) relative to the first augmented reality element 806. When the mouth is within a specified proximity range of the first augmented reality element 806 or the first augmented reality element 806 covers at least a portion of the mouth, the augmentation component 230 can modify the presentation of the first augmented reality element 806 by changing one or more of the color and size of the first augmented reality element 806. The augmentation component 230 can modify the color by darkening the first augmented reality element 806, or can change the size by reducing one or more dimensions of the first augmented reality element 806 so that the first augmented reality element 806 is depicted as being within a certain range. Fig.10 Or it disappears inside the mouth of the object of interest 802 afterwards.

[0071] In operation 750, augmentation component 230 causes at least one second augmented reality element within the graphical user interface to be presented with the object of interest (e.g., the face). In some embodiments, augmentation component 230 causes presentation of the at least one second augmented reality element based on the proximity determined in operation 730. Augmentation component 230 may cause presentation of the at least one second augmented reality element based on the proximity of operation 730 and in response to the reference feature changing 806 from the first state to the second state.

[0072] In some embodiments, at least one second augmented reality element may be presented within the graphical user interface at a distance from the face and the reference feature. The at least one second augmented reality element may be presented as traveling or moving along a trajectory toward the object of interest or the reference feature. For example, Fig.10 As shown in , upon or after modifying the first augmented reality element 806, the augmentation component 230 can cause the presentation of the second augmented reality element 1010. In some embodiments, the second augmented reality element 1010 can be initially presented in a manner similar to that of the first augmented reality element 806. The second augmented reality element 1010 can initially occupy a different position than the first augmented reality element 806. In some embodiments, the second augmented reality element 808 is presented as traveling along a trajectory or path within the graphical user interface 800. The trajectory or path can be similar or different than the trajectory or path of the first augmented reality element 806. For example, Fig.10 As shown in , second augmented reality element 1010 can be presented as falling along a similar trajectory or direction as first augmented reality element 806 , but initially located in a different portion of graphical user interface 800 .

[0073] In some embodiments, in addition to modifying the presentation of the first augmented reality element, the augmentation component 230 modifies the presentation of the object of interest. For example, where the object of interest is a face, the augmentation component 230 can modify the depiction of the face within the graphical user interface to generate a modified face. In some cases, the modification of the face can be performed in response to movement of a portion of the face and modifying the presentation of the first augmented reality element. The augmentation component 230 can cause the modified face to be presented simultaneously with the presentation of one or more of the first augmented reality element and the at least one second augmented reality element.

[0074] Fig.11 A flow chart illustrating an example method 1100 for controlling a user interface using an object depicting the user interface is depicted in accordance with some example embodiments. The operations of method 1100 may be performed by components of augmented reality system 160. In some cases, certain operations of method 1100 may be performed using one or more operations of method 300 or method 700 or as sub-operations of one or more operations of method 300 or method 700, as described in more detail below. For example, in some embodiments, one or more operations of method 1100 may be performed as part of operation 340, as a sub-operation of operation 340, or in response to the above description of the operation 340. Figure 3 The operation 340 described above is performed.

[0075] In operation 1110, the movement component 240 identifies a change in position of an object of interest within a graphical user interface or within a field of view of an image acquisition component of the client device 110. The change in position can be a change from a first position to a second position. The motion component 240 can identify the change in position in a manner similar to or the same as described above with respect to operation 340. The motion component 240 can identify or detect one or more points on the object of interest within a first frame of the video stream to identify the first position. In subsequent frames, the motion component 240 can identify one or more points on the object of interest and compare the positions of the points in the first frame and the subsequent frames to determine the change in position.

[0076] In operation 1120, the movement component 240 determines an object position of an object of interest within the graphical user interface at a first time or a first reference frame. In embodiments where the object of interest is a face, the movement component 240 can determine a head position within the graphical user interface at the first time. The object position can include at least a portion of the graphical user interface surrounding the object of interest.

[0077] In operation 1130, the motion component 240 determines the spatial position of the first augmented reality element at the second time. In some embodiments, the motion component 240 determines the spatial position of the first augmented reality element in a manner similar or identical to that described above with reference to operations 720 and 730. In determining the spatial position of the first augmented reality element, the movement component 240 or the augmentation component 230 determines the position of at least one point, feature, or aspect of the first augmented reality element within the graphical user interface depicted on the display component of the client device 110.

[0078] In operation 1142, the enhancement component 230 determines the proximity of the spatial position relative to the object position at a second time. The enhancement component 230 can determine that the proximity of the spatial position relative to the object position exceeds a miss threshold. In some embodiments, the enhancement component 230 compares the spatial position to the object position. The enhancement component 230 can compare the positions using one or more values, coordinates, or other suitable digital representations of the spatial position and the object position.

[0079] In some embodiments, the enhancement component 230 may determine that the proximity exceeds the miss threshold by comparing the difference identified between the values ​​for the spatial position and the object position to the miss threshold. In the event that the value for the miss threshold is less than the value for the difference between the spatial position and the object position, the enhancement component 230 may identify the spatial position as a miss. The enhancement component 230 may also determine that the proximity exceeds the miss threshold by identifying a buffer around the spatial position for the first augmented reality element and a buffer around the object position for the object of interest. The enhancement component 230 may determine that the value of the difference between the edge of the buffer for the first augmented reality element and the buffer for the object of interest is greater than the value of the miss threshold. Although described with reference to a specified embodiment, the enhancement component 230 may determine that the proximity exceeds the miss threshold by any suitable method.

[0080] In operation 1144, the augmentation component 230 modifies the presentation of the first augmented reality element by modifying one or more of the size, color, and position of the first augmented reality element over a period of time. Figure 6 and Fig.10 The first augmented reality element may be modified in a manner similar or identical to that described above. The augmentation component 230 may modify any suitable aspect, property, or value of the first augmented reality element corresponding to one or more of the miss threshold, the location of the object of interest, the theme of the game, or any other suitable configuration.

[0081] In operation 1146, the augmentation component 230 terminates the presentation of the first augmented reality element after modifying the presentation of the first augmented reality element. In some embodiments, the augmentation component 230 terminates the presentation of the first augmented reality element before causing the presentation of at least one second augmented reality element. The augmentation component 230 may terminate the presentation of the first augmented reality element at a specified or predetermined amount of time after the modification of the first augmented reality element. In some cases, the augmentation component 230 terminates the presentation of the first augmented reality element based on modifying one or more aspects of the first augmented reality element. For example, in the case where the first augmented reality element flies over the shoulder of the user or an object of interest, the augmentation component 230 may terminate the presentation of the first augmented reality element after modifying the size and color of the first augmented reality element to depict the first augmented reality element as passing through the field of view, passing a specified distance of the object of interest, or any other suitable effect.

[0082] In operation 1152, the enhancement component 230 determines at the second time that the proximity of the spatial position relative to the object position is below a miss threshold. In some embodiments, the enhancement component 230 can determine that the proximity is below the miss threshold in a manner similar or the same as described above with respect to operation 1142. In determining the proximity of the spatial position and the object position, the enhancement component 230 can compare a value for the spatial position and a value for the object position. The value representing the difference between the spatial position and the object position can be determined to be less than or otherwise below the miss threshold.

[0083] In operation 1154, the enhancement component 230 determines the termination position of the first augmented reality element at a third time. In some embodiments, the termination position is within the boundary of the object of interest. In some embodiments, the enhancement component 230 determines the termination position of the first augmented reality element based on the trajectory of the first augmented reality element. The enhancement component 230 can also determine the termination position of the first augmented reality element based on the trajectory of the first augmented reality element and the position of the object of interest (e.g., the object position). For example, in the case where the first augmented reality element will hit or hit the object of interest, based on the value being lower than the miss threshold, the enhancement component 230 can determine the position of the object of interest such as the face, neck, or shoulder where the first augmented reality element will hit. In some embodiments, the enhancement component 230 uses one or more of the vertical position, horizontal position, and depth position of the object of interest to determine the termination point. The enhancement component 230 can also determine the termination time (e.g., the third time) when the first augmented reality element will hit the object of interest based on one or more of the trajectory of the first augmented reality element, the speed of the first augmented reality element, the position of the object of interest, and any other suitable information.

[0084] In operation 1156, the augmentation component 230 modifies the presentation of the first augmented reality element by modifying the first augmented reality element from the first state to the second state. The second state of the first augmented reality element can be maintained at a terminal position on the object of interest for a period of time. In some cases, the first state includes a depiction of the first augmented reality element presented in the previous operation as described herein. The second state can represent the first augmented reality element in response to a simulated impact of the first augmented reality element on the object of interest. For example, in the case where the first augmented reality element is a graphical representation of a tomato, the first state can depict the tomato as intact and undamaged. The second state can depict the tomato as crushed, bruised, or otherwise deformed in response to the impact of the first augmented reality element on the object of interest. In this example, the first augmented reality element depicted in the second state (e.g., a splashed tomato) can be presented in a graphical user interface within a predetermined time period (e.g., 1 to 5 seconds), a time period during which the first augmented reality element moves (e.g., a splashed tomato slides down the user's face), or any other suitable time period.

[0085] Fig.12 A flow chart illustrating an example method 1200 for controlling a user interface using an object depicting a user interface is depicted in accordance with some example embodiments. The operations of method 1200 may be performed by components of augmented reality system 160. In some cases, certain operations of method 1200 may be performed using one or more operations of methods 300, 700, or 900 or as sub-operations of one or more operations of methods 300, 700, or 900, as will be described in more detail below. For example, in some embodiments, one or more operations of method 1200 may be performed as part of operation 350, as a sub-operation of operation 350, or in response to the above description of the operation 350. Figure 3 The operation 350 described above is performed.

[0086] In operation 1210, the augmentation component 230 causes the presentation of a first augmented reality element traveling in a first direction (e.g., a first trajectory) relative to the object of interest. In some embodiments, the augmentation component 230 is configured as described above with respect to Figure 3 and Figure 8 The augmented component 230 may initially present the first augmented reality element at a first location within the graphical user interface. The augmented reality element may iteratively or continuously change the position of the first augmented reality element at intervals, rates, or other mechanisms to depict the first augmented reality element as moving along a path of travel or track within the graphical user interface and at a selected speed or rate.

[0087] In operation 1220, one or more of the enhancement component 230 and the motion component 240 determine contact between the first augmented reality element and the area of ​​the object of interest. In some embodiments, the contact may be determined in response to the movement of a portion of the object of interest. In some embodiments, the enhancement component 230 may determine the contact in a manner similar to or identical to that described above with respect to operation 1152. In this embodiment, the enhancement component 230 may determine the contact based on the point of the first augmented reality element and the point of the object of interest being within a specified proximity. In some embodiments, the enhancement component 230 determines the contact by identifying a first edge of the first augmented reality element and a second edge of the object of interest. For example, the first edge and the second edge may be identified as a pixel barrier, a generated outline, or any other appropriately defined intersection between the first augmented reality element and the object of interest, respectively, and another aspect depicted within a graphical user interface. The enhancement component 230 may determine the contact by identifying the location and time at which a portion of the first edge and a portion of the second edge will be rendered at the same location, coordinates, pixels, or other defined locations.

[0088] In operation 1230, augmentation component 230 causes presentation of the first augmented reality element traveling in a second direction relative to the object of interest. In some embodiments, augmentation component 230 modifies the trajectory (e.g., traveling in a first direction to traveling in a second direction) in response to the contact determined in operation 1220. For example, where the first augmented reality element is depicted as a ball and the object of interest is a face, augmentation component 230 may modify the trajectory of the ball (e.g., toward the ball) based on simulated contact of the ball and the face. In some embodiments, augmentation component 230 modifies the trajectory by changing one or more angles of the trajectory (e.g., bouncing at an angle relative to the initial path of travel). Augmentation component 230 may also modify the trajectory by depicting the first augmented reality element as traveling along the same path of travel as in operation 1210 in a direction opposite to the initial direction of the path of travel (e.g., a reverse route).

[0089] In operation 1240, augmentation component 230 causes presentation of at least one second augmented reality element. The at least one second augmented reality element can be similar to the first augmented reality element (e.g., another ball, tomato, or candy), or can be different from the first augmented reality element. For example, the at least one second augmented reality element can be a celebratory graphic, a total score, a total points, a score, a word, an effect (e.g., a firework burst, flames, snowflakes, etc.), or any other suitable graphical element within the graphical user interface and within the video stream.

[0090] In some embodiments, operation 1240 includes one or more sub-operations. For example, in operation 1242, augmentation component 230 causes presentation of an augmented reality element that represents contact between the first augmented reality element and the area of ​​the object of interest. In such embodiments, the augmented reality element representing contact can be a graphical representation of the contact. In such cases, the augmented reality element can be temporary or short-lived, appearing for a short period of time and then ceasing to be presented. For example, the augmented reality element representing contact can be a set of lines indicating a collision, a word (e.g., an onomatopoeia such as "bonk"), or any other suitable representation.

[0091] In operation 1244, the augmented component 230 causes the presentation of an augmented reality element that represents the interaction history of the face relative to the first augmented reality element. In some embodiments, the augmented component 230 generates an effect on the object of interest, such as simulated damage, bruises, blood, swelling, transferred paint, scratches, dents, or other representations of the interaction history. The augmented reality element representing the interaction history can remain unchanged for the duration of the use session. In some cases, the augmented reality element representing the interaction history is presented for a predetermined or dynamic time period. In the case where the time period is dynamic, the time period can be based on the length of the interaction history (e.g., the amount of time the session has been active), the characteristics of the first augmented reality element (e.g., size, shape, hardness, or other aspects of the element), multiple interactions of the object of interest with the first augmented reality element, or any other suitable aspect or feature of one or more of the first augmented reality element, the object of interest, and the user session.

[0092] Modules, components, and logic

[0093] Certain embodiments are described herein as including a logic or multiple components, modules, or mechanisms. Components may constitute hardware components. A "hardware component" is a tangible unit that is capable of performing certain operations and may be configured or arranged in some physical manner. In various example embodiments, a computer system (e.g., a stand-alone computer system, a client computer system, or a server computer system) or a hardware component of a computer system (e.g., at least one hardware processor, a processor, or a group of processors) is configured by software (e.g., an application or application portion) as a hardware component for performing certain operations as described herein.

[0094] In some embodiments, the hardware components are implemented mechanically, electronically, or in any suitable combination thereof. For example, the hardware components may include a dedicated circuit or logic that is permanently configured to perform certain operations. For example, the hardware components may be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware components may also include a programmable logic or circuit that is temporarily configured by software to perform certain operations. For example, the hardware components may include software contained in a general purpose processor or other programmable processor. It is understood that the decision to implement the hardware components mechanically in a dedicated and permanently configured circuit or in a temporarily configured circuit (e.g., configured by software) may be driven by cost and time considerations.

[0095] Thus, the phrase "hardware component" should be understood to include a tangible entity that is physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain manner or perform certain operations described herein. As used herein, a "hardware-implemented component" refers to a hardware component. Considering embodiments in which hardware components are temporarily configured (e.g., programmed), it is not necessary to configure or instantiate each of the hardware components at any one time. For example, in the case where the hardware component includes a general-purpose processor that is configured by software to become a special-purpose processor, the general-purpose processor can be configured as a corresponding different special-purpose processor (e.g., including different hardware components) at different times. Thus, software can configure a specific one or more processors, for example, to constitute a specific hardware component at one time, and to constitute a different hardware component at a different time.

[0096] Hardware components can provide information to other hardware components and receive information from them. Therefore, the described hardware components can be considered as being communicatively coupled. In the case of multiple hardware components being present at the same time, communication can be achieved by signal transmission (e.g., by appropriate circuits and buses) between or among two or more hardware components. In embodiments where multiple hardware components are configured or instantiated at different times, communication between these hardware components can be achieved, for example, by storing and obtaining information in a memory structure accessible to multiple hardware components. For example, a hardware component performs an operation and stores the output of the operation in a memory device communicatively coupled thereto. Then, another hardware component can access the memory device later to obtain and process the stored output. The hardware component can also initiate communication with an input or output device and can operate on a resource (e.g., an information set).

[0097] The various operations of the example methods described herein may be performed, at least in part, by a processor that is temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such a processor constitutes a processor-implemented component that operates to perform the operations or functions described herein. As used herein, a "processor-implemented component" refers to a hardware component that is implemented using a processor.

[0098] Similarly, the methods described herein may be implemented at least in part by a processor, wherein a particular processor or processors are examples of hardware. For example, at least some of the operations of the method may be performed by a processor or a component implemented by the processor. In addition, the processor may also operate to support the performance of related operations in a "cloud computing" environment or as a "software as a service" (SaaS). For example, at least some of the operations may be performed by a group of computers (as an example of a machine including a processor), which may be accessed via a network (e.g., the Internet) and via an appropriate interface (e.g., an application program interface (API)).

[0099] The performance of certain operations can be distributed among processors, not only residing in a single machine, but also deployed across multiple machines. In some example embodiments, the processor or processor-implemented components are located in a single geographic location (e.g., in a home environment, an office environment, or a server farm). In other example embodiments, the processor or processor-implemented components are distributed across multiple geographic locations.

[0100] application

[0101] Fig.13 Execution of a mobile operating system (e.g., IOS TM ANDROID TM , In one embodiment, the mobile device 1300 includes a touch screen operable to receive tactile data from a user 1302. For example, the user 1302 may physically touch 1304 the mobile device 1300, and in response to the touch 1304, the mobile device 1300 may determine tactile data, such as a touch location, a touch force, or a gesture action. In various example embodiments, the mobile device 1300 displays a home screen 1306 (e.g., iOS 11 or other mobile operating system). TM 1306, which is operable to launch applications or otherwise manage various aspects of mobile device 1300. In some example embodiments, home screen 1306 provides status information such as battery life, connectivity, or other hardware status. User 1302 can activate user interface elements by touching the area occupied by the corresponding user interface element. In this way, user 1302 interacts with applications of mobile device 1300. For example, touching the area occupied by a specific icon included in home screen 1306 causes the application corresponding to the specific icon to be launched.

[0102] like Fig.13As shown in , the mobile device 1300 may include an imaging device 1308. The imaging device 1308 may be a camera or any other device coupled to the mobile device 1300 that is capable of capturing a video stream or one or more continuous images. The imaging device 1308 may be triggered by the augmented reality system 160 or a selectable user interface element to initiate the capture of a video stream or a continuum of images and pass the video stream or the continuum of images to the augmented reality system 160 for processing according to one or more methods described in the present disclosure.

[0103] Many types of applications (also referred to as "application software") may be executed on the mobile device 1300, such as native applications (e.g., TM An application written in Objective-C, Swift, or another suitable language that runs on Android TM The mobile device 900 may include a messaging application, an audio recording application, a camera application, a book reader application, a media application, a fitness application, a file management application, a location application, a browser application, a settings application, a contact application, a phone call application, or other application (e.g., a game application, a social networking application, a biometric monitoring application). In another example, the mobile device 1300 may include a user interface such as a web browser, ... The social messaging application software 1310, which is consistent with some embodiments, allows users to exchange short messages including media content. In this example, the social messaging application software 1310 can incorporate aspects of the embodiments described herein. For example, in some embodiments, the social messaging application includes a short-term media gallery created by the user's social messaging application. These galleries may include videos or pictures posted by the user and viewable by the user's contacts (e.g., "friends"). Alternatively, a public gallery may be created by an administrator of the social messaging application, which includes media from any user of the application (and accessible by all users). In another embodiment, the social messaging application may include a "magazine" feature that includes articles and other content generated by the publisher on the platform of the social messaging application and accessible by any user. Any of these environments or platforms can be used to implement the concepts of the subject matter of the present invention.

[0104] In some embodiments, the short message transmission system may include a message with a short video clip or image that is deleted after a deletion trigger event such as viewing time or viewing completion. In such embodiments, the device implementing the augmented reality system 160 may allow the graphical user interface or the elements depicted therein to be controlled using objects captured by the image acquisition device and presented in the graphical user interface when controlling the interaction. The augmented reality system 160 may allow the graphical user interface to be controlled while generating the short message, and allow the short message to be transmitted to another device using the short message system.

[0105] Software Architecture

[0106] Fig.14 is a block diagram 1400 illustrating the architecture of software 1402 that may be installed on the above-described apparatus. Fig.14 This is merely a non-limiting example of a software architecture, and it will be appreciated that many other architectures may be implemented to facilitate the functionality described herein. In various embodiments, software 1402 is comprised of, for example, Fig.15 1402, which includes a processor 1510, a memory 1530, and an I / O component 1550. In this example architecture, the software 1402 can be conceptualized as a stack of layers, each of which can provide specific functionality. For example, the software 1402 includes layers such as an operating system 1404, a library 1406, a framework 1408, and an application 1410. In operation, consistent with some embodiments, the application 1410 calls an application programming interface (API) call 1412 through the software stack and receives a message 1414 in response to the API call 1412.

[0107] In various embodiments, the operating system 1404 manages hardware resources and provides public services. The operating system 1404 includes, for example, a kernel 1420, services 1422, and drivers 1424. Consistent with some embodiments, the kernel 1420 acts as an abstraction layer between the hardware and other software layers. For example, the kernel 1420 provides functions such as memory management, processor management (e.g., scheduling), component management, network connections, and security settings. Services 1422 can provide other public services to other software layers. According to some embodiments, the drivers 1424 are responsible for controlling the underlying hardware or interfacing with the underlying hardware. For example, the drivers 1424 may include a display driver, a camera driver, drives, flash drives, serial communications drivers (such as Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0108] In some embodiments, the library 1406 provides a low-level common infrastructure utilized by the application 1410. The library 1406 may include a system library 1430 (e.g., a C standard library), which may provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the library 1406 may include an API library 1432, such as a media library (e.g., a library that supports the rendering and manipulation of various media formats such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework for rendering two-dimensional (2D) and three-dimensional (3D) graphics content on a display), a database library (e.g., SQLite providing various relational database functions), a web library (e.g., WebKit providing web browsing functions), etc. The library 1406 may also include a variety of other libraries 1434 to provide many other APIs to the application 1410 .

[0109] According to some embodiments, the framework 1408 provides a high-level common architecture that can be utilized by the applications 1410. For example, the framework 1408 provides various graphical user interface (GUI) functions, high-level resource management, high-level location nodes, etc. The framework 1408 can provide other wide-ranging APIs that can be utilized by the applications 1410, some of which can be specific to a particular operating system or platform.

[0110] In an example embodiment, applications 1410 include a home application 1450, a contacts application 1452, a browser application 1454, a book reader application 1456, a location application 1458, a media application 1460, a messaging application 1462, a game application 1464, and other broadly classified applications such as third-party applications 1466. According to some embodiments, applications 1410 are programs that perform functions defined in the program. Applications 1410 structured in various ways can be created using various programming languages, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In a specific example, third-party applications 1466 (e.g., applications created by entities other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) can be used to develop applications on mobile operating systems such as IOS TM ANDROID TM , PHONE or other mobile operating system). In this example, the third-party application 1466 can call the API call 1412 provided by the operating system 1404 to perform the functions described herein.

[0111] Example machine architecture and machine-readable medium

[0112] Fig.15 1 is a block diagram illustrating components of a machine 1500 capable of reading instructions (e.g., processor-executable instructions) from a machine-readable medium (e.g., a non-transitory machine-readable storage medium) and performing any of the methods discussed herein in accordance with some embodiments. Specifically, Fig.15 A schematic diagram of a machine 1500 in the example form of a computer system is shown, within which instructions 1516 (e.g., software, programs, applications, applets, applications, or other executable code) for causing the machine 1500 to perform any of the methods discussed herein may be executed. In alternative embodiments, the machine 1500 operates as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1500 may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1500 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a network device, a network router, a network switch, a network bridge, or any machine capable of executing instructions 1516 that specifies, in a continuous or otherwise manner, the actions to be taken by the machine 1500. Further, while a single machine 1500 is illustrated, the term "machine" shall also be taken to include any collection of machines 1500 that individually or jointly execute instructions 1516 to perform any of the methodologies discussed herein.

[0113] In various embodiments, machine 1500 includes a processor 1510, a memory 1530, and I / O components 1550 that may be configured to communicate with each other via bus 1502. In an example embodiment, processor 1510 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a composite instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) includes, for example, processor 1512 and processor 1514 that may execute instructions 1516. The term "processor" is intended to include a multi-core processor that may include two or more independent processors (also referred to as "cores") that may execute instructions 1516 simultaneously. Although Fig.15 Multiple processors 1510 are shown, but the machine 1500 may include a single processor having a single core, a single processor having multiple cores (eg, a multi-core processor), multiple processors having single cores, multiple processors having multiple cores, or any combination thereof.

[0114] According to some embodiments, the memory 1530 includes a main memory 1532, a static memory 1534, and a storage unit 1536 accessible to the processor 1510 via the bus 1502. The storage unit 1536 may include a machine-readable medium 1538 on which are stored instructions 1516 embodying any of the methods or functions described herein. The instructions 1516 may also reside, completely or at least partially, within the main memory 1532, within the static memory 1534, within at least one of the processors 1510 (e.g., within a cache memory of the processor), or any suitable combination during execution thereof by the machine 1500. Thus, in various embodiments, the main memory 1532, the static memory 1534, and the processor 1510 are considered to be machine-readable media 1538.

[0115] As used herein, the term "memory" refers to a machine-readable medium 1538 capable of storing data temporarily or permanently, and may be considered to include, but is not limited to, random access memory (RAM), read-only memory (ROM), cache, flash memory, and cache. Although the machine-readable medium 1538 is shown as a single medium in the example embodiment, the term "machine-readable medium" should be considered to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) capable of storing instructions 1516. The term "machine-readable medium" may also be considered to include any medium or combination of multiple media capable of storing instructions (e.g., instructions 1516) for execution by a machine (e.g., machine 1500), such that the instructions, when executed by a processor (e.g., processor 1510) of machine 1500, cause machine 1500 to perform any of the methods described herein. Therefore, "machine-readable medium" refers to a single storage device or apparatus, as well as a "cloud-based" storage system or storage network including multiple storage devices or apparatuses. Thus, the term "machine-readable medium" may be taken to include, but is not limited to, data storage repositories in the form of solid-state memory (e.g., flash memory), optical media, magnetic media, other non-volatile memory (e.g., erasable programmable read-only memory (EPROM)), or any suitable combination thereof. The term "machine-readable medium" specifically excludes non-statutory signals per se.

[0116] I / O components 1550 include a variety of components for receiving input, providing output, generating output, sending information, exchanging information, collecting measurements, etc. In general, it is understood that I / O components 1550 may include Fig.15 Many other components not shown in the figure. The I / O components 1550 are grouped according to function, which is only used to simplify the following discussion, and the grouping is by no means limiting. In various example embodiments, the I / O components 1550 include output components 1552 and input components 1554. The output components 1552 include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), auditory components (e.g., speakers), tactile components (e.g., vibration motors), other signal generators, etc. The input components 1554 include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric inputs, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens that provide the location and force of touch or touch gestures, or other tactile input components), audio input components (e.g., microphones), etc.

[0117] In some further example embodiments, the I / O component 1550 includes, among various other components, a biometric component 1556, a motion component 1558, an environmental component 1560, or a position component 1562. For example, the biometric component 1556 includes a component that detects expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or mouth postures), measures biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), identifies a person (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 1558 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The environment component 1560 includes, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., a thermometer that detects ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., a microphone that detects background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor component (e.g., a machine olfactory detection sensor, a gas detection sensor for detecting dangerous gas concentrations for safety or measuring pollutants in the atmosphere), or other components that may provide indications, measurements, or signals corresponding to the surrounding physical environment. The location component 1562 includes a positioning sensor component (e.g., a global positioning system (GPS) receiver component), an altitude sensor component (e.g., an altimeter, or a barometer that detects air pressure from which altitude can be derived), an orientation sensor component (e.g., a magnetometer), and the like.

[0118] Communication can be achieved using a variety of technologies. I / O components 1550 may include a communication component 1564 that is operable to couple machine 1500 to network 1580 or device 1570 via coupler 1582 and coupler 1572, respectively. For example, communication component 1564 includes a network interface component or another suitable device that interfaces with network 1580. In other examples, communication component 1564 includes a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, Components (e.g., low power ), Components and other communication components that provide communication via other modes. Device 1570 may be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via a universal serial bus (USB)).

[0119] In addition, in some embodiments, the communication component 1564 detects an identifier or includes a component operable to detect an identifier. For example, the communication component 1564 includes a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional bar codes such as universal product codes (UPC) bar codes, multi-dimensional bar codes such as quick response (QR) codes, Aztec codes, data matrix, digital graphics, Max codes, PDF417, Super codes, Uniform Commercial Code Reduced Space Symbols (UCC RSS)-2D bar codes, and other optical codes), an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag), or any suitable combination thereof. In addition, various information can be derived via the communication component 1564 that can indicate a particular location, such as a location via an Internet Protocol (IP) geographic location, a location via a location information (such as a location information), ..., a location information, a location information, a location information, a location information, a location information, a location information, a location information, a location information, a location information, a location information, a location information, a location information, a location information, a location information, a location information, a location information, a location information, a location information, a location information, a location information, a location information, a location information, a location information, a location information, a location information, a location information Signal triangulation location, via detection Or the location of an NFC beacon signal, etc.

[0120] Transmission medium

[0121] In various example embodiments, the portion of network 1580 may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a portion of the Internet, a portion of a public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, 1580, or a portion of network 1580, may include a wireless or cellular network, and coupling 1582 may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or another type of cellular or wireless coupling. In this example, coupling 1582 may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) networks, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other standards defined by various standards development organizations, other remote protocols, or other data transmission technologies.

[0122] In an example embodiment, instructions 1516 are sent or received over network 1580 using a transmission medium via a network interface device (e.g., a network interface component included in communication component 1564), and utilize any of a number of well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, in other example embodiments, instructions 1516 are sent or received to device 1570 using a transmission medium via coupling 1572 (e.g., a peer-to-peer coupling). The term "transmission medium" may be taken to include any intangible medium capable of storing, encoding, or carrying instructions 1516 for execution by machine 1500, and includes digital or analog communication signals or other intangible media to facilitate the communication of such software implementations.

[0123] Furthermore, because the machine-readable medium 1538 does not embody a propagating signal, the machine-readable medium 1538 can be non-transient (in other words, without any short-lived signals). However, marking the machine-readable medium 1538 as "non-transient" should not be interpreted to mean that the medium cannot be moved. The medium should be considered to be transportable from one physical location to another. Additionally, because the machine-readable medium 1538 is tangible, the medium can be considered a machine-readable device.

[0124] language

[0125] Throughout the specification, multiple instances can be implemented as components, operations or structures described as single instances. Although the individual operations of the method are shown and described as individual operations, individual operations can be performed simultaneously, and the operations do not need to be performed in the order shown. The structure and function presented as the individual components in the example configuration can be implemented as a combined structure or component. Similarly, the structure and function presented as a single component can be implemented as a plurality of components separated. These and other variations, modifications, additions and improvements fall within the scope of this paper theme.

[0126] Although an overview of the inventive subject matter has been described with reference to specific example embodiments, various modifications and changes may be made to these embodiments without departing from the broader scope of the embodiments of the present disclosure. Such embodiments of the inventive subject matter may be referred to herein individually or collectively by the term "invention", which is merely for convenience and is not intended to limit the scope of the present application to any single disclosure or inventive concept if, in fact, more than one is disclosed.

[0127] The embodiments shown herein are described in sufficient detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and derived therefrom, so that structural and logical substitutions and changes may be made without departing from the scope of the present disclosure. Therefore, the specific embodiments should not be considered restrictive, and the scope of the various embodiments is limited only by the appended claims and the full scope of equivalents granted by these claims.

[0128] As used herein, the term "or" may be interpreted in an inclusive or exclusive manner. In addition, multiple instances may be provided as a single instance for the resources, operations, or structures described herein. In addition, the boundaries between various resources, operations, components, engines, and data stores are somewhat arbitrary, and specific operations are shown in the context of specific illustrative configurations. Other allocations of functions may be envisioned, and these other allocations may fall within the scope of various embodiments of the present disclosure. Typically, structures and functions presented as separate resources in an example configuration may be implemented as a combined structure or resource. Similarly, structures and functions presented as a single resource may be implemented as separate resources. These and other changes, modifications, additions, and improvements fall within the scope of the embodiments of the present disclosure as represented by the appended claims. Therefore, the specification and drawings are considered to be illustrative rather than restrictive.

Claims

1. A method for augmented reality user interface control, comprising: initiating, by one or more processors of a device, an augmented reality mode configured to present an augmented reality element within a graphical user interface presented on a display component of the device; detecting a face within a field of view of an image acquisition component coupled to the device; In response to detecting the face, sequentially presenting within the graphical user interface a set of augmented reality elements such that a subset of augmented reality elements and the face are simultaneously depicted within the graphical user interface, wherein a first augmented reality element in the subset of augmented reality elements is presented at a varying position on the graphical user interface in successively acquired and presented frames; detecting movement of a reference feature of the face relative to the first augmented reality element, the movement of the reference feature being detected by identifying a first state of the reference feature and determining a change of the reference feature from the first state to a second state, wherein the first state corresponds to a first position of a set of landmark points corresponding to the reference feature in a first frame of the graphical user interface and the second state corresponds to a second position of at least one landmark point in the set of landmark points in a second frame of the graphical user interface; determining a facial location of the reference feature in the second frame; determining a spatial position of the first augmented reality element within the graphical user interface in the second frame; determining a proximity of the spatial location to the facial location as being within a proximity threshold; and In response to determining that the proximity of the spatial location to the facial location is within a proximity threshold, modifying presentation of the first augmented reality element and causing presentation of at least one second augmented reality element in the subset of augmented reality elements.

2. The method according to claim 1, wherein: The reference feature is a mouth, the first state is an open state of the mouth, and the second state is a closed state of the mouth, and modifying the presentation of the first augmented reality element further comprises: determining that the first augmented reality element covers at least a portion of the mouth in the first state at the spatial position, the portion of the mouth covered by the first augmented reality element being an inner surface of the mouth; In response to the first augmented reality element covering the portion of the mouth, modifying presentation of the first augmented reality element for a period of time before terminating presentation of the first augmented reality element; and Causing presentation of the at least one second augmented reality element spaced a distance from the face and the reference feature, the at least one second augmented reality element moving along a trajectory toward the face.

3. The method according to claim 1, wherein: Detecting the movement of the face further comprises: identifying within the graphical user interface a change in position of the face from a first position to a second position; In response to identifying the second position, determining a head position within the graphical user interface at a first time; and The spatial position of the first augmented reality element is determined at a second time.

4. The method according to claim 3, further comprising: determining that the proximity of the spatial position relative to the head position at the second time exceeds a miss threshold; modifying presentation of the first augmented reality element by modifying one or more of a size, a color, and a position of the first augmented reality element over a period of time; as well as Presentation of the first augmented reality element is terminated after modifying presentation of the first augmented reality element and before causing presentation of the at least one second augmented reality element.

5. The method according to claim 3, further comprising: determining that the proximity of the spatial position relative to the head position at the second time is below a miss threshold; determining, at a third time, a termination position of the first augmented reality element, the termination position being within a boundary of the face; as well as The presentation of the first augmented reality element is modified by modifying the first augmented reality element from a first state to a second state, the second state of the first augmented reality element being maintained for a period of time at the terminal location on the face.

6. The method according to claim 1, wherein: Modifying the presentation of the first augmented reality element further comprises: causing presentation of the first augmented reality element traveling in a first direction relative to the face; responsive to the movement of the reference feature of the face, determining contact between the first augmented reality element and an area of ​​the face; and In response to the contact, presentation of the first augmented reality element is caused to travel in a second direction relative to the face.

7. The method according to claim 6, wherein: Causing presentation of the at least one second augmented reality element further comprises: causing presentation of an augmented reality element representing the contact between the first augmented reality element and the area of ​​the face; and Causing presentation of an augmented reality element representing an interaction history of the face with respect to the first augmented reality element.

8. The method according to claim 1, further comprising: responsive to the movement of the reference feature of the face and modifying the presentation of the first augmented reality element, modifying the depiction of the face within the graphical user interface to generate a modified face; and Causing presentation of the modified face concurrently with presentation of one or more of the first augmented reality element and the at least one second augmented reality element.

9. A system for augmented reality user interface control, comprising: one or more processors; as well as a non-transitory processor-readable storage medium coupled to the one or more processors, the non-transitory processor-readable storage medium storing processor-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: initiating, by the one or more processors, an augmented reality mode configured to present augmented reality elements within a graphical user interface presented on a display component of the device; detecting, by the one or more processors, a face within a field of view of an image acquisition component coupled to the device; In response to detecting the face, sequentially presenting, by the one or more processors, a set of augmented reality elements within the graphical user interface such that a subset of augmented reality elements and the face are simultaneously depicted within the graphical user interface, wherein a first augmented reality element in the subset of augmented reality elements is presented at a varying position on the graphical user interface in successively acquired and presented frames; detecting, by the one or more processors, movement of a reference feature of the face relative to the first augmented reality element, the movement of the reference feature being detected by identifying a first state of the reference feature and determining a change of the reference feature from the first state to a second state, wherein the first state corresponds to a first position of a set of landmark points corresponding to the reference feature in a first frame of the graphical user interface and the second state corresponds to a second position of at least one landmark point in the set of landmark points in a second frame of the graphical user interface; determining a facial location of the reference feature in the second frame; determining a spatial position of the first augmented reality element within the graphical user interface in the second frame; determining a proximity of the spatial location to the facial location as being within a proximity threshold; and In response to determining that the proximity of the spatial location to the facial location is within a proximity threshold, the one or more processors modify the presentation of the first augmented reality element and cause presentation of at least one second augmented reality element in the subset of augmented reality elements.

10. The system according to claim 9, wherein: Detecting the movement of the face further comprises: identifying within the graphical user interface a change in position of the face from a first position to a second position; In response to identifying the second position, determining a head position within the graphical user interface at a first time; and The spatial position of the first augmented reality element is determined at a second time.

11. The system according to claim 9, wherein: Modifying the presentation of the first augmented reality element further comprises: causing presentation of the first augmented reality element traveling in a first direction relative to the face; responsive to the movement of the reference feature of the face, determining contact between the first augmented reality element and an area of ​​the face; and In response to the contact, presentation of the first augmented reality element is caused to travel in a second direction relative to the face.

12. A non-transitory processor-readable storage medium storing processor-executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising: initiating, by one or more processors of a device, an augmented reality mode configured to present an augmented reality element within a graphical user interface presented on a display component of the device; detecting, by the one or more processors, a face within a field of view of an image acquisition component coupled to the device; In response to detecting the face, sequentially presenting, by the one or more processors, a set of augmented reality elements within the graphical user interface such that a subset of augmented reality elements and the face are simultaneously depicted within the graphical user interface, wherein a first augmented reality element in the subset of augmented reality elements is presented at a varying position on the graphical user interface in successively acquired and presented frames; detecting, by the one or more processors, movement of a reference feature of the face relative to the first augmented reality element, the movement of the reference feature being detected by identifying a first state of the reference feature and determining a change of the reference feature from the first state to a second state, wherein the first state corresponds to a first position of a set of landmark points corresponding to the reference feature in a first frame of the graphical user interface and the second state corresponds to a second position of at least one landmark point in the set of landmark points in a second frame of the graphical user interface; determining a facial location of the reference feature in the second frame; determining a spatial position of the first augmented reality element within the graphical user interface in the second frame; determining a proximity of the spatial location to the facial location as being within a proximity threshold; and In response to determining that the proximity of the spatial location to the facial location is within a proximity threshold, the one or more processors modify the presentation of the first augmented reality element and cause presentation of at least one second augmented reality element in the subset of augmented reality elements.

13. The non-transitory processor-readable storage medium of claim 12, wherein: Detecting the movement of the face further comprises: identifying within the graphical user interface a change in position of the face from a first position to a second position; In response to identifying the second position, determining a head position within the graphical user interface at a first time; and The spatial position of the first augmented reality element is determined at a second time.

14. The non-transitory processor-readable storage medium of claim 12, wherein: Modifying the presentation of the first augmented reality element further comprises: causing presentation of the first augmented reality element traveling in a first direction relative to the face; responsive to the movement of the reference feature of the face, determining contact between the first augmented reality element and an area of ​​the face; and In response to the contact, presentation of the first augmented reality element is caused to travel in a second direction relative to the face.

Images