Method, device, and non-transitory computer-readable storage medium for displaying augmented reality content

Abstract

This disclosure relates to methods, apparatus, and non-transitory computer-readable storage media for displaying augmented reality content. The method includes receiving voice input to select augmented reality content for display along with an image; determining at least one keyword included in the voice input; determining that the at least one keyword indicates an action to be performed on the image; determining a first attribute of an object depicted in the image; assigning weights to each of the first attributes of the object; ranking a plurality of augmented reality content items based on the assigned weights and a second attribute of the action; selecting the highest-ranking augmented reality content item from the plurality of augmented reality content items based on the ranking; and activating the highest-ranking augmented reality content item for the image.

Description

[0001] This application is a divisional application of patent application No. 202180031101.3 entitled "Voice-based selection of augmented reality content for detected objects", filed on March 25, 2021, with international application number PCT / US2021 / 024191, and entered the Chinese national phase on October 26, 2022.

[0002] Cross-references to related applications

[0003] This patent application claims priority to U.S. Application Serial No. 17 / 211,321, filed March 24, 2021, which claims the benefit of U.S. Provisional Patent Application No. 63 / 000,071, filed March 26, 2020, entitled “SPEECH-BASED SELECTION OF AUGMENTED REALITY CONTENT FOR DETECTED OBJECTS,” which is incorporated herein by reference in its entirety. This patent application also claims priority to U.S. Application Serial No. 17 / 211,158, filed March 24, 2021, which claims the benefit of U.S. Provisional Patent Application No. 63 / 000,056, filed March 26, 2020, entitled “SPEECH-BASED SELECTION OF AUGMENTED REALITY CONTENT,” which is incorporated herein by reference in its entirety. Technical Field

[0004] This disclosure generally relates to messaging applications, including providing the display of augmented reality content within messaging applications. Background Technology

[0005] Messaging systems facilitate the exchange of message content between users. For example, a messaging system allows a user to exchange message content (e.g., text, images) with one or more other users. Summary of the Invention

[0006] According to one aspect of this disclosure, a method for displaying augmented reality content is provided, comprising: receiving voice input to select augmented reality content for display along with an image; determining at least one keyword included in the voice input; determining that the at least one keyword indicates an action to be performed on the image; determining a first attribute of an object depicted in the image; assigning weights to each of the first attributes of the object; ranking a plurality of augmented reality content items based on the assigned weights and a second attribute of the action; selecting the highest-ranked augmented reality content item from the plurality of augmented reality content items based on the ranking; and activating the highest-ranked augmented reality content item for the image.

[0007] According to another aspect of this disclosure, an apparatus is provided, comprising: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations including: receiving voice input to select augmented reality content for display together with an image; determining at least one keyword included in the voice input; determining that the at least one keyword indicates an action to be performed on the image; determining a first attribute of an object depicted in the image; assigning weights to each of the first attributes of the object; ranking a plurality of augmented reality content items based on the assigned weights and a second attribute of the action; selecting the highest-ranked augmented reality content item from the plurality of augmented reality content items based on the ranking; and activating the highest-ranked augmented reality content item for the image.

[0008] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided, the computer-readable storage medium including instructions that, when executed by a computer, cause the computer to perform operations including: receiving voice input to select augmented reality content for display together with an image; determining at least one keyword included in the voice input; determining that the at least one keyword indicates an action to be performed for the image; determining a first attribute of an object depicted in the image; assigning weights to each of the first attributes of the object; ranking a plurality of augmented reality content items based on the assigned weights and a second attribute of the action; selecting the highest-ranked augmented reality content item from the plurality of augmented reality content items based on the ranking; and activating the highest-ranked augmented reality content item for the image. Attached Figure Description

[0009] To facilitate the identification of any particular element or action in discussion, one or more highest-order digits in the figure reference numerals indicate the figure number in which the element was first introduced.

[0010] Figure 1 It is a graphical representation of a networked environment in which the present disclosure can be deployed, according to some example implementations.

[0011] Figure 2 It is a graphical representation of a message-transmitting client application based on some example implementations.

[0012] Figure 3 It is a graphical representation of a data structure maintained in a database, based on some example implementations.

[0013] Figure 4 It is a graphical representation of a message based on some example implementations.

[0014] Figure 5 This is a flowchart of an access restriction process based on some example implementations.

[0015] Figure 6 This is an interactive diagram illustrating the process of selecting augmented reality content based on voice input to perform actions on objects, according to some example implementations.

[0016] Figures 7A to 7B A user interface for selecting augmented reality content based on voice input to provide object recognition information is shown, according to some example implementations.

[0017] Figures 8A to 8B The diagram illustrates a user interface for selecting augmented reality content based on voice input to solve a problem corresponding to an object, according to some example implementations.

[0018] Figure 9 This is a flowchart illustrating a process for selecting augmented reality content based on voice input to perform actions on objects, according to some exemplary embodiments.

[0019] Figure 10 This is a block diagram illustrating a software architecture in which the present disclosure can be implemented according to some example embodiments.

[0020] Figure 11 It is a graphical representation of a machine in the form of a computer system according to some example embodiments, within which a set of instructions can be executed to cause the machine to perform any or more of the methods discussed herein. Detailed Implementation

[0021] Messaging systems typically allow users to exchange content items (e.g., messages, images, and / or videos) with each other within a messaging thread. Messaging systems can be implemented in conjunction with augmented reality systems or otherwise function to display augmented reality content related to the messaging. For example, when creating message content, augmented reality content is combined with image data captured by the device's camera. However, users may desire convenient creation and / or selection of augmented reality content related to messaging.

[0022] The disclosed implementations provide a messaging application running on a device to select between augmented reality content items based on received voice input (e.g., corresponding to applying an augmented reality experience or camera lens). The voice input corresponds to a voice command for performing an action on an object depicted in a captured image. The messaging application determines keywords from the voice input, for example, by requesting keywords from a voice recognition service.

[0023] The messaging application identifies keywords indicating objects depicted in an image and also instructs on actions to be performed on those objects. The application sends the object's attributes and the action's attributes to a server configured to select augmented reality content items (e.g., corresponding to augmented reality experiences) based on these attributes. The application then displays the augmented reality content corresponding to the selected item, along with the captured image data.

[0024] Figure 1 This is a block diagram illustrating an example messaging system 100 for exchanging data (e.g., messages and associated content) via network 108. Messaging system 100 includes instances of client devices 102, each hosting multiple applications including a messaging client application 104 and a speech recognition client application 106. Each messaging client application 104 is communicatively coupled to other instances of the messaging client application 104 and a messaging server system 112 via network 108 (e.g., the Internet).

[0025] Messaging client application 104 can communicate and exchange data with other messaging client applications 104 and messaging server system 112 via network 108. The data exchanged between messaging client applications 104 and between other messaging client applications 104 and messaging server system 112 includes functions (e.g., commands that call functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0026] Communication between the disclosed messaging client application 104 and the speech recognition client application 106 can be transmitted directly. Alternatively or additionally, communication between the disclosed messaging client application 104 and the speech recognition client application 106 can be transmitted indirectly (e.g., via one or more servers).

[0027] In one or more embodiments, the voice recognition client application 106 is a separate and distinct application from the messaging client application 104. For example, the voice recognition client application 106 is downloaded and installed by the client device 102 separately from (e.g., before or after) the messaging client application 104. Furthermore, the voice recognition client application 106 is provided by a different entity or organization than the entity or organization that provides the messaging client application 104. In one or more embodiments, the voice recognition client application 106 is an application that can be accessed by the client device 102 using different login credentials than the messaging client application 104. For example, the voice recognition client application 106 may maintain a first user account, and the messaging client application 104 may maintain a second user account.

[0028] In one or more alternative embodiments, the speech recognition client application 106 is a component included as part of the messaging client application 104. For example, the speech recognition client application 106 includes one or more hardware and / or software components integrated within the messaging client application 104. In this way, the messaging client application 104, combined with the speech recognition server system 128 and / or the messaging server system 112, is configured to perform the functions of the speech recognition client application 106.

[0029] In one or more embodiments, the voice recognition client application 106 is configured to perform voice recognition on audio input (e.g., corresponding to a user's voice) received at the client device 102. For example, the audio input is received by the messaging client application 104 and corresponds to a voice command for performing a specific function (e.g., displaying augmented reality content).

[0030] In one or more embodiments, the speech recognition client application 106 corresponds to a client-side component (e.g., via network 108) communicating with the speech recognition server system 128, which corresponds to a server-side component for performing speech recognition. For example, the speech recognition client application 106 is configured to receive audio input provided by a messaging client application 104, determine that the audio input includes a trigger word for activating speech recognition, and request the speech recognition server system 128 to perform speech recognition on additional audio input (e.g., a voice command corresponding to the remainder of the audio input after the trigger word or to subsequent audio input received by the messaging client application 104).

[0031] In some implementations, messaging client application 104 (e.g., upon startup) activates the camera of client device 102. Messaging client application 104 allows a user to request a scan of one or more items in a camera feed captured by the camera. For example, messaging client application 104 may receive a user selection of dedicated scan options (e.g., buttons) presented along with the camera feed. In alternative implementations, messaging client application 104 may detect physical contact between a user's finger and an area of ​​the touchscreen for a threshold time period. For example, messaging client application 104 determines that a user touches the screen and holds their finger on the screen for more than three seconds. In response, messaging client application 104 captures an image displayed on the screen and processes the image to identify one or more objects in the image. Alternatively or additionally, as described herein, a scanning operation for detecting objects may be performed in response to voice input (e.g., a voice command) to perform an action targeting the object. In some implementations, messaging client application 104 uses one or more trained classifiers and / or environmental factors to identify objects in the image.

[0032] Messaging server system 112 provides server-side functionality to a specific messaging client application 104 via network 108. While some functions of messaging system 100 are described herein as being performed by messaging client application 104 or by messaging server system 112, it will be understood that the location of certain functions within messaging client application 104 or messaging server system 112 is a design choice. For example, it may technically be preferred to initially deploy certain technologies and functions within messaging server system 112, but later migrate those technologies and functions to messaging client application 104, where client device 102 has sufficient processing power.

[0033] The messaging server system 112 supports various services and operations provided to the messaging client application 104. Such operations include sending data to and receiving data from the messaging client application 104, and processing data generated by the messaging client application 104. As examples, this data may include message content, client device information, graphical elements, geolocation information, media annotations and overlays, virtual objects, message content persistence conditions, social network information, and live event information. Data exchange within the messaging system 100 is invoked and controlled via functions available through the user interface (UI) (e.g., a graphical user interface) of the messaging client application 104.

[0034] Now, specifically to message server system 112, API server 110 (Application Programming Interface server) is coupled to application server 114 and provides a programming interface to application server 114. Application server 114 is communicatively coupled to database server 118, which facilitates access to database 124, which stores data associated with messages processed by application server 114.

[0035] Specifically, API server 110 handles the receiving and sending of message data (e.g., commands and message payloads) between client device 102 and application server 114. Specifically, API server 110 provides a set of interfaces (e.g., routines and protocols) that can be invoked or queried by messaging client application 104 to call functions of application server 114. API server 110 exposes various functions supported by application server 114, including account registration; login functionality; sending messages from one messaging client application 104 to another messaging client application 104 via application server 114; sending media files (e.g., graphical elements, images, or videos) from messaging client application 104 to messaging server application 116, and making them accessible to another messaging client application 104; a list of graphical elements; setting up collections of media data (e.g., stories); retrieving such collections; retrieving the friend list of the user of client device 102; maintaining augmented reality content items; retrieving messages and content; adding and removing friends to the social graph; the location of friends within the social graph; accessing user conversation data; accessing avatar information stored on messaging server system 112; and opening application events (e.g., related to messaging client application 104).

[0036] Application server 114 hosts multiple applications and subsystems, including messaging server application 116, image processing system 120, social networking system 122, and augmented reality system 126. Messaging server application 116 implements numerous messaging techniques and functions, particularly involving the aggregation and other processing of content (e.g., text and multimedia content) received from multiple instances of messaging client application 104. As will be described in more detail, text and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or libraries). These collections are then made available to 10 by messaging server application 116. Given the hardware requirements for such processing, messaging server application 116 may also perform additional processor- and memory-intensive data processing on the server side.

[0037] Application server 114 also includes an image processing system 120 dedicated to performing various image processing operations, typically for images or videos received within the payload of messages at message delivery server application 116. In one or more implementations, a portion of the image processing system 120 may also be implemented by augmented reality system 126.

[0038] Social networking system 122 supports various social networking functions and services, and makes these functions and services available to messaging server application 116. To this end, social networking system 122 maintains and accesses an entity graph within database 124. Examples of functions and services supported by social networking system 122 include identifiers of other users of messaging system 100 with whom a particular user has a relationship or who "follows" them, as well as identifiers of other entities and users of interest to a particular user. Such other users can be referred to as the user's friends. Social networking system 122 can access location information associated with each of the user's friends to determine where they live or are currently geographically located. Social networking system 122 can maintain a location profile for each of the user's friends, indicating the geographical location where the user's friends reside.

[0039] The messaging client application 104 includes a set of functions that allow the client device 102 to access the augmented reality system 126. The augmented reality system 126 generates and maintains a list of augmented reality content items. These augmented reality content items can correspond to augmented reality experiences used to supplement captured image data with augmented reality content.

[0040] In one or more embodiments, the augmented reality system 126 provides one or more attributes for determining (e.g., receiving) one or more attributes of an object (e.g., name) and / or one or more actions performed on the object (e.g., providing identification information, providing visual effects, providing solutions). The augmented reality system 126 provides a search for one or more augmented reality content items (e.g., virtual objects) associated with one or more attributes of the object and / or action, and ranks the virtual objects (e.g., based on association and weights assigned to each attribute). The augmented reality system 126 renders one or more virtual objects or graphical elements of the highest-ranking augmented reality content item over the captured image.

[0041] Application server 114 is communicatively coupled to database server 118, which facilitates access to database 124, which stores data associated with messages processed by messaging server application 116. Database 124 may be a third-party database. For example, application server 114 may be associated with a first entity, and database 124, or a portion thereof, may be associated with and hosted by a second, different entity. In some implementations, database 124 stores user data collected by the first entity regarding each of the various users of the services provided by the first entity. For example, user data includes usernames, phone numbers, passwords, addresses, friends, activity information, preferences, videos or content consumed by the user, etc.

[0042] Figure 2 This is a block diagram illustrating further details of a messaging system 100 according to an example embodiment. Specifically, the messaging system 100 is shown to include a messaging client application 104 and an application server 114, which in turn includes several subsystems, namely a short-timer system 202, a collection management system 204, and an annotation system 206.

[0043] The short-lived timer system 202 is responsible for implementing short-lived access to content permitted by the messaging client application 104 and the messaging server application 116. To this end, the short-lived timer system 202 combines multiple timers that selectively display messages and associated content, and enable access to messages and associated content via the messaging client application 104, based on durations and display parameters associated with messages or sets of messages (e.g., stories). Further details regarding the operation of the short-lived timer system 202 are provided below.

[0044] The collection management system 204 is responsible for managing collections of media (e.g., collections of text, image, video, and audio data). In some examples, collections of content (e.g., messages, including images, videos, text, and audio) can be organized into “event libraries” or “event stories.” Such collections can be made available for a specified time period (e.g., the duration of the event the content relates to). For example, content related to a concert can be made available as a “story” for the duration of the concert. The collection management system 204 can also be responsible for publishing icons that notify the user interface of the messaging client application 104 of the existence of a specific collection.

[0045] The collection management system 204 also includes a curation interface 208 that allows collection managers to manage and curate specific collections of content. For example, the curation interface 208 enables event organizers to curate collections of content related to a specific event (e.g., removing inappropriate content or redundant messages). Additionally, the collection management system 204 employs machine vision (or image recognition technology) and content rules to automatically curate content collections. In some implementations, users may be paid compensation to include user-generated content in the collection. In such cases, the curation interface 208 operates to automatically pay such users for using their content.

[0046] Annotation system 206 provides various functions that enable users to annotate or otherwise modify or edit media content associated with messages. For example, annotation system 206 provides functions related to generating and publishing media overlays for messages processed by messaging system 100. Annotation system 206 can operablely supply media overlays or supplements (e.g., image filtering) to messaging client application 104 based on the geographic location of client device 102. In another example, annotation system 206 can operablely supply media overlays to messaging client application 104 based on other information (e.g., the social network information of the user of client device 102). Media overlays can include audio and visual content as well as visual effects. Examples of audio and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects can be applied to content items (e.g., photos) at client device 102. For example, media overlays can include text that can be overlaid on a photograph taken by client device 102. In another example, media overlays include location identifiers (e.g., Venice Beach) overlays, names of live events, or business names (e.g., beach cafes) overlays. In another example, annotation system 206 uses the geographic location of client device 102 to identify media overlays that include the name of a merchant at the geographic location of client device 102. The media overlays may include other tags associated with the merchant. The media overlays may be stored in database 124 and accessed through database server 118.

[0047] In one example implementation, annotation system 206 provides a user-based publishing platform that allows users to select geographic locations on a map and upload content associated with those locations. Users can also specify environments in which particular media overlays should be provided to other users. Annotation system 206 generates media overlays that include the uploaded content and associates the uploaded content with the selected geographic location.

[0048] In another example implementation, annotation system 206 provides a merchant-based publishing platform that enables merchants to select specific media coverage associated with geographic locations through a bidding process. For example, annotation system 206 associates the media coverage of the highest bidder with the corresponding geographic location within a predefined time period.

[0049] Figure 3 This is a schematic diagram illustrating a data structure 300 that can be stored in a database 124 of a message sending server system 112 according to some example embodiments. Although the contents of the database 124 are shown to include multiple tables, it should be understood that the data can be stored in other types of data structures (e.g., as an object-oriented database).

[0050] Database 124 includes message data stored in message table 314. Entity table 302 stores entity data, including entity diagram 304. Entities whose records are maintained in entity table 302 can include individuals, company entities, organizations, objects, locations, events, etc. Regardless of type, any entity in the message delivery server system 112 with respect to its stored data can be an identified entity. Each entity is assigned a unique identifier and an entity type identifier (not shown).

[0051] Entity Graph 304 stores information about the relationships and associations between entities. As an example only, such relationships can be social relationships based on interests or activities, or professional relationships (e.g., working in a common company or organization).

[0052] Message table 314 may store a collection of conversations between a user and one or more friends or entities. Message table 314 may include various attributes for each conversation, such as a list of participants, the size of the conversation (e.g., the number of users and / or the number of messages), the chat color of the conversation, a unique identifier for the conversation, and any other conversation-related characteristics.

[0053] Database 124 also stores annotation data in the form of filter examples in annotation table 312. Database 124 also stores the annotated content received in annotation table 312. The data of the filters stored in annotation table 312 are associated with and applied to videos (whose data is stored in video table 310) and / or images (whose data is stored in image table 308). In one example, the filter is an overlay displayed as an overlay on an image or video during presentation to the recipient user. Filters can be of various types, including user-selected filters from a library of filters presented to the sending user by messaging client application 104 when the sending user is composing a message. Other types of filters include geolocation filters (also known as geographic filters), which can be presented to the sending user based on geographic location. For example, based on geographic location information determined by the Global Positioning System (GPS) unit of client device 102, messaging client application 104 can present neighborhood-specific or location-specific geographic filters within the UI. Another type of filter is a data filter, which can be selectively presented to the sending user by the messaging client application 104 based on other input or information collected by the client device 102 during the message creation process. Examples of data filters include the current temperature at a specific location, the current speed of the sending user, the battery life of the client device 102, or the current time.

[0054] Other annotation data that can be stored in image table 308 are augmented reality content items (e.g., corresponding to augmented reality experiences or shots). Augmented reality content items can be real-time special effects and sounds that can be added to images or videos.

[0055] As described above, augmented reality content items, overlays, image transformations, AR images, and similar terms refer to modifications that can be made to a video or image. This includes real-time modifications, which modify an image as it is captured using the device's sensors and then display the modified image on the device's screen. It also includes modifications to stored content, such as video clips in a library that can be modified. For example, in a device that accesses multiple augmented reality content items, a user can use a single video clip with multiple augmented reality content items to see how different augmented reality content items will modify the stored clip. For example, by selecting different augmented reality content items for the content, multiple augmented reality content items applying different pseudo-random motion models can be applied to the same content. Similarly, real-time video capture can be used with the shown modifications to demonstrate how the video image currently captured by the device's sensors will modify the captured data. Such data can simply be displayed on the screen and not stored in memory, or the content captured by the device's sensors can be recorded and stored in memory with or without modification (or both). In some systems, preview features can show how different augmented reality content items look simultaneously in different windows on the display. This can, for example, make it possible to view multiple windows with different pseudo-random animations on the display simultaneously.

[0056] Therefore, data, and various systems that use augmented reality content items or other such transformation systems to modify content using that data, can involve: the detection of objects (e.g., faces, hands, bodies, cats, dogs, surfaces, objects, etc.); tracking such objects as they leave or enter the field of view in a video frame and move around the field of view; and modifying or transforming such objects while they are being tracked. In various implementations, different methods can be used to implement such transformations. For example, some implementations may involve generating a three-dimensional mesh model of one or more objects and using transformations and animated textures of the model within the video to implement the transformation. In other implementations, tracking points on the object can be used to place an image or texture (which may be two-dimensional or three-dimensional) at the tracked location. In yet another implementation, neural network analysis of video frames can be used to place images, models, or textures within content (e.g., images or frames of a video). Thus, augmented reality content items involve both images, models, and textures used to create transformations within the content, and additional modeling and analysis information required to implement such transformations using object detection, tracking, and placement.

[0057] Real-time video processing can be performed using any type of video data (e.g., video streams, video files, etc.) stored in the memory of any type of computerized system. For example, a user can load video files and store them in the device's memory, or the device's sensors can be used to generate video streams. Additionally, computer-animated models can be used to process any object, such as a human face and body parts, animals, or inanimate objects (e.g., chairs, cars, or other objects).

[0058] In some implementations, when a specific modification is selected along with the content to be transformed, the computing device identifies the elements to be transformed and then detects and tracks them if they exist in the video frames. The elements of the object are modified according to the modification request, thereby transforming the frames of the video stream. The transformation of the video stream frames can be performed using different methods for different types of transformations. For example, for a transformation of frames that primarily involves changing the form of object elements, feature points are calculated for each element of the object (e.g., using an Active Shape Model (ASM) or other known methods). A feature-point-based mesh is then generated for each element of at least one of the object's elements. This mesh is used for subsequent stages of tracking the elements of the object in the video stream. During tracking, the aforementioned mesh for each element is aligned with the position of each element. Additional points are then generated on the mesh. A first set of first points is generated for each element based on the modification request, and a second set of points is generated for each element based on the first set of points and the modification request. The frames of the video stream can then be transformed by modifying the elements of the object based on the first and second sets of points and the mesh. In such a method, the background of the modified object can also be changed or distorted by tracking and modifying the background.

[0059] In one or more embodiments, transformations that alter some regions of an object using its elements can be performed by calculating feature points for each element of the object and generating a mesh based on the calculated feature points. Points are generated on the mesh, and then various regions are generated based on these points. The elements of the object are then tracked by aligning the regions for each element with the positions for each of at least one element, and the frames of the video stream can be transformed by modifying the characteristics of the regions based on a request for modification. Depending on the specific request for modification, the characteristics of the mentioned regions can be transformed in different ways. Such modifications can involve: changing the color of the region; removing at least some portions of the region from the frames of the video stream; including one or more new objects into the region based on the request for modification; and modifying or distorting the elements of the region or object. In various embodiments, any combination of such modifications or other similar modifications can be used. For certain models to be animated, some feature points can be selected as control points for the entire state space used to determine options for animatening the model.

[0060] In some implementations of computer animation models that use face detection to transform image data, a specific face detection algorithm (e.g., Viola-Jones) is used to detect faces in the image. Then, an Active Shape Modeling (ASM) algorithm is applied to the facial regions of the image to detect facial feature reference points.

[0061] In other implementations, other methods and algorithms suitable for face and / or object detection can be used. For example, in some implementations, features are located using landmarks representing distinguishable points present in most of the images considered. For example, for a face landmark, the location of the left pupil could be used. Secondary landmarks can be used if the initial landmark is not recognizable (e.g., if the person is wearing an eye patch). Such a landmark recognition process can be used for any such object. In some implementations, the set of landmarks forms a shape. The shape can be represented as a vector using the coordinates of the points in the shape. One shape is aligned with another shape using a similarity transformation (allowing translation, scaling, and rotation) that minimizes the average Euclidean distance between the points of the shape. The average shape is the average of the aligned training shapes.

[0062] In some implementations, the search begins with a landmark search based on an average shape aligned with the position and size of the face determined by a global face detector. This search then repeats the following steps: proposing provisional shapes by adjusting the positions of shape points using template matching of the image texture around each point, and then conforming the provisional shapes to a global shape model until convergence occurs. In some systems, individual template matching is unreliable, and the shape model aggregates the results of weak template matchers to form a stronger overall classifier. The entire search is repeated at each level of the image pyramid, from coarse to fine resolution.

[0063] The transformation system can be implemented by capturing image or video streams on a client device (e.g., client device 102) and performing complex image manipulations locally on client device 102 while maintaining an appropriate user experience, computation time, and power consumption. Complex image manipulations can include size and shape changes, emotion transfer (e.g., changing a face from frowning to smiling), state transfer (e.g., aging an object, reducing its apparent age, or changing its gender), style transfer, application of graphical elements, and any other suitable image or video manipulation implemented by a convolutional neural network that has been configured to execute efficiently on client device 102.

[0064] In some example implementations, a computer animation model for transforming image data can be used by a system in which a user can capture an image or video stream (e.g., a selfie) using a client device 102 having a neural network operating as part of a messaging client application 104 operating on the client device 102. A transformation system operating within the messaging client application 104 determines the presence of objects (e.g., faces) within the image or video stream and provides a modification icon associated with the computer animation model for transforming the image data, or the computer animation model may exist in association with the interface described herein. The modification icon includes changes that may be used to modify the user's face within the image or video stream as part of a modification operation. Once a modification icon is selected, the transformation system initiates a process of transforming the user's image to reflect the selected modification icon (e.g., generating a smiling face on the user). In some implementations, once an image or video stream is captured and a specified modification is selected, the modified image or video stream can be presented in a graphical user interface displayed on a mobile client device. The transformation system may implement a complex convolutional neural network on a portion of the image or video stream to generate and apply the selected modification. In other words, users can capture image or video streams, and once an edit icon is selected, the modified result can be presented in real-time or near real-time. Furthermore, the modifications can be persistent while the video stream is captured and the selected edit icon continues to toggle. Machine-trained neural networks can be used to achieve such modifications.

[0065] In some implementations, the graphical user interface (GUI) presenting the modifications performed by the transformation system may offer the user additional interactive options. Such options may be based on the interface used to initiate the selection and content capture of a specific computer animation model (e.g., initiated from a content creator user interface). In various implementations, the modifications may be persistent after the initial selection of the modification icon. The user can toggle the modification on or off by tapping or otherwise selecting the face being modified by the transformation system and save it for later viewing or browsing to other areas of the imaging application. In cases where the transformation system modifies multiple faces, the user can globally toggle the modification on or off by tapping or selecting a single face modified and displayed within the GUI. In some implementations, individual faces within a group of multiple faces can be modified individually, or such modifications can be toggled individually by tapping or selecting individual faces or a series of individual faces displayed within the GUI.

[0066] As mentioned above, video table 310 stores video data, which in one embodiment is associated with messages whose records are maintained within message table 314. Similarly, image table 308 stores image data associated with messages whose message data is stored in entity table 302. Entity table 302 can associate various annotations from annotation table 312 with various images and videos stored in image table 308 and video table 310.

[0067] Augmented Reality Content Item Table 316 stores indications (e.g., a list) of augmented reality content items that can be selected and activated by the messaging client application 104. In one or more embodiments, each augmented reality content item in Augmented Reality Content Item Table 316 is associated with one or more object attributes. Each augmented reality content item in Augmented Reality Content Item Table 316 may also be associated with one or more predefined terms (e.g., using metadata tags, names, etc.). In one or more embodiments, the messaging client application 104 searches the object attributes and / or predefined terms stored in Augmented Reality Content Item Table 316 to identify one or more augmented reality content items associated with scanned objects or objects identified in captured images. Each augmented reality content item stored in Augmented Reality Content Item Table 316 includes one or more graphical elements or virtual objects that may or may not be animated. Each augmented reality content item also includes instructions on where to position the graphical elements or virtual objects relative to other objects depicted in the captured image.

[0068] Story table 306 stores data about collections of messages and associated image, video, or audio data, compiled into collections (e.g., stories or libraries). The creation of a specific collection can be initiated by a specific user (e.g., each user whose records are maintained in entity table 302). Users can create "personal stories" in the form of collections of content that have already been created and sent / broadcast by that user. For this purpose, the UI of messaging client application 104 may include icons that can be selected by the user to allow the sending user to add specific content to his or her personal story.

[0069] Collections can also constitute "live stories," which are collections of content from multiple users created manually, automatically, or using a combination of manual and automatic technologies. For example, a "live story" can constitute a curated stream of user-submitted content from different locations and events. Users whose client devices have location services enabled and are at a common location event at a specific time can be presented with options to contribute content to a specific live story, for example, via the UI of messaging client application 104. The messaging client application 104 can identify live stories to users based on their location. The end result is a "live story" told from a community perspective.

[0070] Another type of content collection is called a "location story," which allows users whose client devices 102 are located in a specific geographic location (e.g., at a university or on a university campus) to contribute to a specific collection. In some implementations, contributing to a location story may require a second level of authentication to verify that the end user belongs to a specific organization or other entity (e.g., is a student on a university campus).

[0071] Figure 4 This is a schematic diagram illustrating the structure of a message 400 according to some embodiments, generated by a messaging client application 104 for transmission to another messaging client application 104 or a messaging server application 116. The content of a particular message 400 is used to populate a message table 314 stored in a database 124 accessible to the messaging server application 116. Similarly, the content of the message 400 is stored in memory as “in transit” or “in flight” data of the client device 102 or application server 114. The message 400 is shown to include the following components:

[0072] ●Message Identifier 402: A unique identifier that identifies message 400.

[0073] ●Message text payload 404: The text to be generated by the user via the user interface of the client device 102 and included in message 400.

[0074] ●Message Image Payload 406: Image data captured by the camera component of the client device 102 or retrieved from the memory component of the client device 102 and included in message 400.

[0075] ● Message video payload 408: Video data captured by the camera component or retrieved from the memory component of the client device 102 and included in message 400.

[0076] ●Message audio payload 410: Audio data captured by the microphone or retrieved from the memory component of the client device 102 and included in message 400.

[0077] ● Message annotation 412: Annotation data (e.g., filters, stickers or other enhancements) representing annotations to be applied to message image payload 406, message video payload 408 or message audio payload 410 of message 400.

[0078] ● Message Duration Parameter 414: A parameter value, in seconds, indicating the amount of time that the content of the message (e.g., message image payload 406, message video payload 408, message audio payload 410) will be presented or made accessible to the user via the messaging client application 104.

[0079] ● Message geolocation parameter 416: Geographic location data (e.g., latitude and longitude coordinates) associated with the message's content payload. Multiple message geolocation parameter values ​​416 may be included in the payload, each of which is associated with a content item included in the content (e.g., a specific image within the message image payload 406 or a specific video within the message video payload 408).

[0080] ● Message Story Identifier 418: An identifier value that identifies one or more sets of content (e.g., "story") associated with a specific content item in the message image payload 406 of message 400. For example, multiple images within the message image payload 406 may each be associated with multiple sets of content using their own identifier values.

[0081] ● Message Tag 420: Each message 400 can be labeled with multiple tags, each tag indicating the subject of the content included in the message payload. For example, in the case where a specific image included in the message image payload 406 depicts an animal (e.g., a lion), the tag value can be included within the message tag 420 indicating the relevant animal. The tag value can be manually generated based on user input, or it can be automatically generated using, for example, image recognition.

[0082] ●Message sender identifier 422: An identifier (e.g., a messaging system identifier, email address, or device identifier) ​​indicating the user of the client device 102 on which message 400 is generated and from which message 400 is sent.

[0083] ● Message Recipient Identifier 424: An identifier (e.g., a messaging system identifier, email address, or device identifier) ​​indicating the user of the client device 102 to which message 400 is addressed.

[0084] The content (e.g., values) of each component of message 400 can be pointers to locations in tables storing content data values. For example, image values ​​in message image payload 406 can be pointers to locations (or addresses) within image table 308. Similarly, values ​​in message video payload 408 can point to data stored in video table 310, values ​​in message annotation 412 can point to data stored in annotation table 312, values ​​in message story identifier 418 can point to data stored in story table 306, and values ​​in message sender identifier 422 and message receiver identifier 424 can point to user records stored in entity table 302.

[0085] Figure 5 This is a schematic diagram illustrating an access restriction process 500, according to which access to content (e.g., a short message 502 and a multimedia payload of associated data) or a collection of content (e.g., a short message group 504) can be time-restricted (e.g., short-lived).

[0086] A brief message 502 is shown to be associated with a message duration parameter 506, the value of which determines the amount of time the messaging client application 104 will display the brief message 502 to the receiving user. In one implementation, depending on the amount of time specified by the sending user using the message duration parameter 506, the receiving user may view the brief message 502 for up to 10 seconds.

[0087] Message duration parameter 506 and message receiver identifier 424 are shown as inputs to message timer 512, which is responsible for determining the amount of time for which brief message 502 will be shown to a specific receiving user identified by message receiver identifier 424. Specifically, brief message 502 will only be shown to the relevant receiving user within the time period determined by the value of message duration parameter 506. Message timer 512 is shown as providing an output to a more generalized brief timer system 202, which is responsible for the overall timing of displaying content (e.g., brief message 502) to the receiving user.

[0088] Figure 5 The transient message 502 shown is included within a transient message group 504 (e.g., a collection of messages in a personal story or event story). The transient message group 504 has an associated group duration parameter 508, the value of which determines the duration for which the transient message group 504 is presented and accessible to a user of the messaging system 100. For example, the group duration parameter 508 could be the duration of a concert, where the transient message group 504 is a collection of content belonging to that concert. Alternatively, when setting up and creating the transient message group 504, the user (owner user or curator user) can specify the value of the group duration parameter 508.

[0089] Additionally, each short message 502 within a short message group 504 has an associated group participation parameter 510, the value of which determines the duration for which the short message 502 is accessible within the context of the short message group 504. Therefore, a particular short message group 504 can "expire" and become inaccessible within its context before the short message group 504 itself expires according to the group duration parameter 508. The group duration parameter 508, the group participation parameter 510, and the message receiver identifier 424 each provide input to a group timer 514, which is operable to first determine whether a particular short message 502 of the short message group 504 will be displayed to a specific receiving user, and if so, determine for how long. Note that, as a result of the message receiver identifier 424, the short message group 504 also knows the identity of the specific receiving user.

[0090] Therefore, the group timer 514 operatively controls the total usage period of the associated ephemeral message group 504 and the individual ephemeral messages 502 included in the ephemeral message group 504. In one embodiment, each ephemeral message 502 within the ephemeral message group 504 remains viewable and accessible for a period of time specified by the group duration parameter 508. In another embodiment, a ephemeral message 502 may expire within the context of the ephemeral message group 504 based on the group participation parameter 510. Note that even within the context of the ephemeral message group 504, the message duration parameter 506 can still determine the duration for which a particular ephemeral message 502 is displayed to the receiving user. Therefore, the message duration parameter 506 determines the duration for which a particular ephemeral message 502 is displayed to the receiving user, regardless of whether the receiving user views the ephemeral message 502 within or outside the context of the ephemeral message group 504.

[0091] The short-lived timer system 202 can also operatively remove a specific short-lived message 502 from the short-lived message group 504 based on the determination that the associated group participation parameter 510 has expired. For example, when the sending user has established a group participation parameter 510 for 24 hours from the date of publication, the short-lived timer system 202 will remove the relevant short-lived message 502 from the short-lived message group 504 after the specified 24 hours. The short-lived timer system 202 also operates to remove the short-lived message group 504 when the group participation parameter 510 for each short-lived message 502 within the short-lived message group 504 has expired, or when the short-lived message group 504 itself has expired according to the group duration parameter 508.

[0092] In certain use cases, the creator of a specific ephemeral message group 504 can specify an indefinite group duration parameter 508. In this case, the expiration of the group participation parameter 510 for the last remaining ephemeral message 502 within the ephemeral message group 504 will determine when the ephemeral message group 504 itself expires. In this case, adding a new ephemeral message 502 with a new group participation parameter 510 to the ephemeral message group 504 effectively extends the lifetime of the ephemeral message group 504 to a value equal to the group participation parameter 510.

[0093] In response to the short-lived timer system 202 determining that a short-lived message group 504 has expired (e.g., is no longer accessible), the short-lived timer system 202 communicates with the messaging system 100 (and, in particular, the messaging client application 104) to cause the markers (e.g., icons) associated with the relevant short-lived message group 504 to no longer be displayed in the user interface of the messaging client application 104. Similarly, when the short-lived timer system 202 determines that the message duration parameter 506 for a particular short-lived message 502 has expired, the short-lived timer system 202 causes the messaging client application 104 to no longer display the markers (e.g., icons or text labels) associated with the short-lived message 502.

[0094] Figure 6 This is an interactive diagram illustrating a process 600 for selecting augmented reality content based on voice input to perform an action on an object, according to some example implementations. For illustrative purposes, reference is made primarily herein. Figure 1The process 600 is described using a messaging client application 104, a speech recognition client application 106, a speech recognition server system 128, and a messaging server system 112. However, process 600 may be executed by one or more other components and / or by other suitable devices. Further, for illustrative purposes, blocks of process 600 are described herein as occurring serially or linearly. However, multiple blocks of process 600 may occur in parallel. Additionally, blocks of process 600 do not need to be executed in the order shown, and / or one or more blocks of process 600 need not be executed and / or may be replaced by other operations.

[0095] As described herein, client device 102 is configured to provide selection of augmented reality content items (e.g., corresponding to augmented reality experiences) based on voice input (e.g., a user's voice) received at messaging client application 104, and also based on objects detected in image data captured by the device's camera. For example, messaging client application 104 requests voice recognition client application 106 to perform voice recognition on the voice input to determine keywords for searching multiple augmented reality content items (e.g., available augmented reality content items). Messaging client application 104 determines that the keywords indicate actions to be performed on objects depicted in the captured image data. Messaging client application 104 sends attributes of the object and attributes of the action to messaging server system 112, which is configured to select augmented reality content items (e.g., augmented reality experiences) based on the attributes. Messaging client application 104 displays the selected augmented reality content item (e.g., augmented reality experience) along with the captured image data.

[0096] At block 602, messaging client application 104 causes the camera of client device 102 to capture images. For example, messaging client application 104 is configured to activate the camera by default when messaging client application 104 is initialized. Alternatively or additionally, messaging client application 104 automatically activates the camera for a specific interface of messaging client application 104 (e.g., an interface for generating multimedia content, e.g., to be included in a message).

[0097] At block 604, messaging client application 104 receives voice input (e.g., a voice command for selecting an augmented reality content item to be displayed). The voice input is received when the camera of client device 102 is activated and image data is captured.

[0098] In one or more embodiments, the voice input received by the messaging client application 104 includes trigger words instructing the initialization of voice recognition (e.g., "Hey," "Hey Messaging App," "Messaging App"), and also includes voice commands for selecting augmented reality content items to be displayed (e.g., as described below regarding...). Figures 7A to 7B and Figures 8A to 8B The discussion includes phrases like "what kind of book is that?" and "solving this mathematical problem." The messaging client application 104 is configured to provide voice input to the speech recognition client application 106 for speech recognition (operation 606).

[0099] In one or more embodiments, the speech recognition client application 106 is configured to process speech input in multiple stages. For example, in a first stage, the speech recognition client application 106 uses speech activity detection (VAD) or other known techniques to detect whether at least a portion of the speech input corresponds to human speech. In some embodiments, the speech recognition client application 106 implements VAD locally, causing the client device 102 to perform human speech detection itself.

[0100] In the second phase, if human speech is detected in the speech input, the speech recognition client application 106 determines locally (e.g., on client device 102) whether the speech input includes predefined trigger words (e.g., "hey", "hey messaging app", "messaging app"). Alternatively, the detection of trigger words can be performed by the speech recognition server system 128 based on a request sent to it.

[0101] In the third stage, if the voice input is determined to include a trigger word, the voice recognition client application 106 provides the voice input to the voice recognition server system 128 to determine keywords based on the voice input. As mentioned above, in addition to including trigger words, the voice input may also include voice commands for selecting a specific augmented reality content item to be displayed (e.g., determining keywords from it).

[0102] It should be noted that the three stages described above correspond to examples of speech recognition, and other technologies may be used in place of these stages or in addition to these stages. For example, as an alternative to requiring a trigger word, speech recognition client application 106 may be configured to initiate the speech recognition service of speech recognition server system 128 in response to a predefined gesture (e.g., user input associated with a hardware button on client device 102 and / or a software button displayed within messaging client application 104). The user gesture may be accompanied by voice input provided by the user, wherein the voice input includes voice commands (and does not include trigger words).

[0103] In another example, the voice input may include a trigger word but initially does not include a voice command (e.g., “Hey,” “Hey messaging app,” and / or “messaging app,” followed by a pause indicating the end of the voice input). In such a case, the voice recognition client application 106, in conjunction with the messaging client application 104, can prompt the user with the voice command (e.g., by displaying a message and / or an audio message, such as “How can I help you?”). The user can respond via the messaging client application 104 using additional voice input corresponding to the voice command, wherein the voice recognition client application 106 is configured to send the additional voice input to the voice recognition server system 128 for voice recognition and keyword determination.

[0104] Therefore, at block 608, the speech recognition client application 106 determines one or more keywords based on the speech input, wherein such determination is based on the speech recognition client application 106 sending the speech input to the speech recognition server system 128. In some embodiments, the speech recognition server system 128 is configured to implement one or more automatic speech recognition (ASR) algorithms or other known techniques to convert the speech input into one or more text-based keywords. The speech recognition client application 106 then sends one or more keywords (e.g., received from the speech recognition server system 128) to the messaging client application 104 (operation 610).

[0105] In one or more embodiments, messaging client application 104 is configured to display a text-based version of the voice command portion of the voice input. In this regard, speech recognition server system 128 may perform ASR to convert the entire voice command portion of the voice input into text, and may provide that text (along with keywords) to the messaging client application (e.g., at operation 610). Alternatively or additionally, in some embodiments, messaging client application 104 is configured to display graphical elements (e.g., animations) when receiving voice input from a user.

[0106] At block 612, messaging client application 104 determines keywords indicating objects depicted in an image captured by the device's camera and also indicates actions to be performed on those objects. For example, messaging client application 104 determines a first keyword indicating an object depicted in the captured image and a second keyword indicating an action to be performed on that object.

[0107] As mentioned above, the messaging client application 104 is configured to implement or otherwise access an object recognition algorithm (e.g., including machine learning algorithms) configured to scan the captured image and detect / track the movement of objects within the image. As a non-limiting example, detectable objects within the image include: faces, human body parts, animals and parts thereof, landscapes, objects in nature, inanimate objects (e.g., chairs, books, cars, buildings, other structures), illustrations of objects (e.g., on posters and / or flyers), text-based objects, equation-based objects, etc. Furthermore, the messaging client application 104 is configured to determine or otherwise access object attributes such as name, type, style, color, size, shape, texture, geographic location, and / or other supplementary information (e.g., song title / artist for objects corresponding to media).

[0108] In one or more implementations, the messaging client application 104 is also configured to determine actions associated with an object. For example, the messaging client application 104 may (e.g., from local and / or remote storage) access one or more corresponding actions associated with an object of a particular type. As a non-limiting example, actions that may be performed on an object include: providing identification and / or supplementary information about the object (e.g., name, style, song title, author, geographic location, etc.), providing a solution to an object corresponding to a problem (e.g., a solution to a mathematical problem, the answer to a question), applying visual effects to the object (e.g., texture, 3D mesh), etc. Furthermore, the messaging client application 104 is configured to determine or otherwise access attributes of the action, such as the type of action (e.g., an action for identification, solving, modification, providing visual effects, etc.), the duration of the action, and / or other characteristics associated with the action.

[0109] Therefore, based on the keywords provided at operation 610, the detection of objects within the captured image data, and the actions associated with different types of objects, the messaging client application 104 is configured to determine that the keywords indicate objects depicted in the image and also indicate actions to be performed on those objects.

[0110] At operation 614, messaging client application 104 sends attributes of an object (e.g., the name of the object) and / or attributes of an action (e.g., the type of the action) to messaging server system 112 (operation 614). In one or more embodiments, messaging client application 104 sends keywords, such as those provided by speech recognition client application 106, as attributes of the object and the action.

[0111] In response, the messaging server system 112 searches for and selects augmented reality content items (e.g., augmented reality experiences) based on the attributes of objects and actions (block 616). In one or more embodiments, the messaging server system 112 is configured to search a set of available augmented reality content items by comparing the attributes of objects and actions with the corresponding attributes and / or predefined terms associated with each of the available augmented reality content items.

[0112] As mentioned above, the augmented reality content item table 316 included in database 124 is configured to specify or tag each augmented reality content item with corresponding attributes and / or predefined terms (e.g., via metadata) to search for augmented reality content items. Therefore, in one or more embodiments, messaging server system 112 queries database 124 with attributes of objects and actions (e.g., the name of the object and the type of action performed), and database 124 can provide one or more selected augmented reality content items as the result of the query.

[0113] Where an object / action attribute corresponds to a corresponding attribute and / or predefined term for more than one augmented reality content item, database 124 can be configured to provide indications of multiple augmented reality content items as the result of a query. Furthermore, database 124 can rank the multiple augmented reality content items (e.g., based on the number of matches and / or the weights assigned to one or more of the object / action attribute, augmented reality content item attributes, and / or predefined terms). Database 124 can provide the ranking indication as part of the query result to messaging server system 112.

[0114] Based on the received results, the messaging server system 112 sends the indication of the selected augmented reality content item, along with the ranking information (if applicable), to the messaging client application 104 (operation 618).

[0115] The messaging client application 104 displays the selected augmented reality content item along with image data captured by the camera of the client device 102 (block 620). In the case where multiple augmented reality content items are provided by the messaging server system 112, the messaging client application 104 can display the highest-ranking augmented reality content item along with the image data, and, for example, provide the remaining augmented reality content items in a carousel in ranking order. See below regarding... Figures 7A to 7B and Figures 8A to 8BThe messaging client application 104, as discussed, displays a carousel interface that allows the user to switch between multiple available augmented reality content items. Available augmented reality content items may include a ranked set of augmented reality content items provided to the messaging client application 104 at operation 618 and / or additional available augmented reality content items (e.g., available augmented reality content items stored in augmented reality content item table 316).

[0116] Therefore, for messaging client application 104, by means of process 600, selection of augmented reality content (e.g., augmented reality content item) can be provided based on voice commands (e.g., voice input) and objects depicted in image data captured by the device's camera. The selected augmented reality content item can be displayed together with the captured image data.

[0117] Figures 7A to 7B User interfaces 702 and 712, according to some example implementations, are shown for selecting augmented reality content (e.g., augmented reality experience) based on voice input to provide object identification information. For example, user interfaces 702 and 712 are displayed within a messaging client application 104. Figure 7A The user interface 702 corresponds to the interface for receiving voice commands, and Figure 7B The user interface 712 corresponds to the interface that applies the selected augmented reality content item to the captured image data.

[0118] exist Figure 7A In one example, user interface 702 includes captured image data 706 corresponding to image data captured by the front-facing camera of client device 102. Alternatively, the image data may be captured by the rear-facing camera of client device 102. In one or more embodiments, the user of button 708 selects to provide messages that generate images (e.g., in response to a press / tap gesture of button 708) and / or videos (e.g., in response to a press and hold gesture of button 708) of screen content, for example, to send to friends, including in stories, etc.

[0119] As discussed above, a user of client device 102 can provide voice input to messaging client application 104. Voice input includes voice commands (e.g., preceded by a trigger word) for selecting specific augmented reality content items. Figure 7A In the example, voice input may include a trigger word (e.g., not shown) followed by a voice command / query of “What book is that?”

[0120] In one or more implementations, while receiving voice input, the messaging client application 104 is configured to display converted text 710, which corresponds to a text-based version of the voice input determined by the speech recognition server system 128. Alternatively or additionally, the messaging client application 104 is configured to display a voice command graphic (not shown), for example, which corresponds to an animation indicating (e.g., real-time) reception of voice input.

[0121] Go to Figure 7B In response to receiving voice input (e.g., by messaging client application 104), user interface 712 is displayed. As discussed above, messaging server system 112 selects augmented reality content items from a plurality of available augmented reality content items based at least in part on keywords from the voice input (e.g., “what”, “book”). For example, the keyword “book” corresponds to an object depicted in captured image data 706 detected by messaging client application 104 as discussed above.

[0122] The messaging client application 104 can detect different objects in the captured image data 706 (e.g., as part of a scanning operation as discussed above). For example, the messaging client application 104 can identify a book object 704 and several other objects (not shown, such as a table, magazine, lamp, etc.) in the captured image data 706. When identifying objects, the messaging client application 104 determines that the keyword "book" is depicted in the captured image data 706 (e.g., as the book object 704).

[0123] Additionally, the messaging client application 104 determines the keyword "what" that indicates the action to be performed on the book object 704. For example, the keyword "what" can be pre-determined to be associated with an action that provides identification information about the object. Other examples of pre-defined keywords associated with actions that provide identification information include, but are not limited to, "identify," "tell me about," "provide information about," etc.

[0124] As discussed above, the messaging server system 112 searches for and selects one or more augmented reality content items based on attributes of the book object 704 and actions (e.g., providing identification information), and provides the messaging client application 104 with shot instructions and / or ranking information. The messaging client application 104 then activates the selected augmented reality content item to display the augmented reality content item 716 (e.g., the title of the book object 704) along with the captured image data 706. For example, the selected augmented reality content item may correspond to an application configured to provide identification information about the book. In one or more implementations, supplementary information element 714 may be displayed to indicate other identification information about the book object 704 (e.g., author, availability). Other identification information may also be provided by the selected augmented reality content item. Furthermore, supplementary information element 714 may correspond to a user-selectable link that redirects to a third-party application for additional information about the book and / or the ability to purchase the book.

[0125] Furthermore, the user interface 712 includes a carousel interface 720 that allows the user to cycle through and / or select different augmented reality content items to apply to the captured image data 706. Each of the available augmented reality content items is represented by an icon that the user can select to switch to the corresponding augmented reality content item. In one or more embodiments, the icon corresponding to the active augmented reality content item (e.g., the selected augmented reality content item icon 718) is displayed in a manner different from (e.g., larger than) the other icons. In one or more embodiments, the user's selection of the selected augmented reality content item icon 718 provides a message for generating images (e.g., in response to a press / tap gesture of the selected augmented reality content item icon 718) and / or videos (e.g., in response to a press and hold gesture of the selected augmented reality content item icon 718), such as for sending to friends, including in stories, etc.

[0126] Figures 8A to 8B User interfaces 802 and 812, according to some example implementations, are shown for selecting augmented reality content (e.g., augmented reality experience) based on voice input to solve a problem corresponding to an object. For example, user interfaces 802 and 812 are displayed within a messaging client application 104. Figure 8A The user interface 802 corresponds to the interface for receiving voice commands, and Figure 8B The user interface 812 corresponds to the interface for applying the selected augmented reality content item to the captured image data.

[0127] exist Figure 8AIn one example, user interface 802 includes captured image data 806 corresponding to image data captured by the front-facing camera of client device 102. Alternatively, the image data may be captured by the rear-facing camera of client device 102. In one or more embodiments, the user selects to provide messages for generating images (e.g., in response to a press / tap gesture of button 808) and / or videos (e.g., in response to a press and hold gesture of button 808) of screen content, for example, to send to friends, including in stories, etc.

[0128] As discussed above, a user of client device 102 can provide voice input to messaging client application 104. Voice input includes voice commands (e.g., preceded by a trigger word) for selecting specific augmented reality content items. Figure 8A In the example, voice input may include a trigger word (e.g., not shown) followed by a voice command / query such as “Solve this math problem”.

[0129] In one or more implementations, while receiving voice input, the messaging client application 104 is configured to display converted text 810, which corresponds to a text-based version of the voice input determined by the speech recognition server system 128. Alternatively or additionally, the messaging client application 104 is configured to display a voice command graphic (not shown), for example, which corresponds to an animation indicating (e.g., real-time) reception of voice input.

[0130] Go to Figure 8B In response to receiving voice input (e.g., by messaging client application 104), user interface 812 is displayed. As discussed above, messaging server system 112 selects augmented reality content items from a plurality of available augmented reality content items based at least in part on keywords from the voice input (e.g., “solve”, “mathematical problem”). For example, the keyword “mathematical problem” corresponds to an object depicted in the captured image data 806 as detected by messaging client application 104 as discussed above.

[0131] The messaging client application 104 can detect different objects in the captured image data 806 (e.g., as part of a scanning operation). For example, the messaging client application 104 can identify a math problem object 804 and several other objects (paper, pen, etc.) in the captured image data 806. When identifying objects, the messaging client application 104 determines that the keyword "math problem" is depicted in the captured image data 806 (e.g., as the math problem object 804).

[0132] Additionally, the messaging client application 104 determines that the keyword "solve" indicates the action to be performed on the mathematical problem object 804. For example, the keyword "solve" can be pre-defined to be associated with the action of solving the problem. Other examples of pre-defined keywords associated with the action of solving the problem include, but are not limited to, "calculate," "derive," and "provide the answer."

[0133] As discussed above, the messaging server system 112 searches for and selects one or more augmented reality content items based on the attributes of the mathematical problem object 804 and the action (e.g., solving a problem), and provides the messaging client application 104 with camera directions and / or ranking information. The messaging client application 104 then activates the selected augmented reality content item to display the augmented reality content item 814 (e.g., the mathematical problem object 804) along with the captured image data 806. For example, the selected augmented reality content item may correspond to an application configured to solve mathematical problems (e.g., a mathematical solver application). In one or more implementations, the messaging client application 104 may provide a user-selectable link (e.g., a button 816) for redirecting to a third-party application that provides additional information about the mathematical problem object 804.

[0134] In addition, the user interface 812 includes a carousel interface 818 that allows the user to cycle through and / or select different augmented reality content items to apply to the captured image data 806. Each of the available augmented reality content items is represented by an icon that the user can select to switch to the corresponding augmented reality content item. In one or more embodiments, the icon corresponding to the active augmented reality content item (e.g., the selected augmented reality content item icon 820) is displayed in a manner different from (e.g., larger than) the other icons. In one or more embodiments, the user's selection of the selected augmented reality content item icon 820 provides a message for generating images (e.g., in response to a press / tap gesture of the selected augmented reality content item icon 820) and / or videos (e.g., in response to a press and hold gesture of the selected augmented reality content item icon 820) to be sent, for example, to friends, included in stories, etc.

[0135] Notice, Figures 7A to 7B and Figures 8A to 8B This corresponds to an example of augmented reality content items that can be selected based on voice input provided by the user (e.g., voice commands). Other examples of augmented reality content item selection are discussed below.

[0136] In one example, messaging client application 104 receives voice input to provide visual effects for a food item depicted in a captured image. For example, the user may have provided a voice command such as "Animate my burger" (or another similar command). In response, messaging client application 104 (e.g., in conjunction with voice recognition client application 106, voice recognition server system 128, and / or messaging server system 112) retrieves an augmented reality content item (e.g., an augmented reality content item) that includes one or more graphical elements that resemble the food item (e.g., an animated image of a burger and / or objects associated with a burger). Messaging client application 104 renders one or more graphical elements around or on top of the food item. One or more graphical elements may be animated to appear to surround the food item.

[0137] In another example, messaging client application 104 receives voice input to provide identification information and / or visual effects of a movie or media item depicted in a captured image. For example, the user may have provided a voice command “What movie is this?” (e.g., or other similar commands). In response, messaging client application 104 (e.g., in conjunction with voice recognition client application 106, voice recognition server system 128, and / or messaging server system 112) retrieves an augmented reality content item that includes information and / or graphic elements associated with the movie or media item. Messaging client application 104 presents the information and / or graphic elements around or on top of the movie or media item. For example, if the cover art includes an image of a lion, the augmented reality content item includes a graphic element that animates the lion, and replaces the image of the lion in the cover art with the animated lion of that graphic element.

[0138] In another example, messaging client application 104 receives voice input to provide weather information for a landscape (e.g., including the sky) depicted in a captured image. For example, the user may have provided a voice command, "What's the weather like here?" (e.g., or another similar command). In response, messaging client application 104 (e.g., in conjunction with voice recognition client application 106, voice recognition server system 128, and / or messaging server system 112) uses (e.g., corresponding to a weather application) augmented reality content item to provide current weather information for the depicted landscape. The current weather information may be based at least in part on the geographic location of client device 102. The selected augmented reality content item provides one or more graphic elements depicting the weather, and messaging client application 104 renders one or more graphic elements over the landscape (e.g., including the sky).

[0139] Figure 9This is a flowchart illustrating a process 900 for selecting augmented reality content based on voice input to perform an action on an object, according to some example implementations. For illustrative purposes, reference is made primarily herein. Figure 1 The process 900 is described using a messaging client application 104, a speech recognition client application 106, a speech recognition server system 128, and a messaging server system 112. However, one or more blocks (or operations) of the process 900 may be performed by one or more other components and / or by other suitable devices. Further for illustrative purposes, blocks of the process 900 are described herein as occurring serially or linearly. However, multiple blocks of the process 900 may occur in parallel. Additionally, blocks of the process 900 do not need to be executed in the order shown, and / or one or more blocks of the process 900 do not need to be executed and / or may be replaced by other operations.

[0140] A messaging client application 104 running on client device 102 causes the camera of client device 102 to capture images (block 902). The messaging client application 104 receives voice input to select augmented reality content (e.g., augmented reality content items) to be displayed along with the images (block 904).

[0141] The messaging client application 104 determines at least one keyword included in the voice input (block 906). Determining at least one keyword may include sending a request to the voice recognition server system 128 to perform voice recognition based on the voice input, and receiving at least one keyword from the voice recognition server system 128 based on the sending request. A first portion of the voice input may include a trigger word, and at least one keyword may be based on a second portion of the voice input excluding the trigger word.

[0142] The messaging client application 104 determines at least one keyword indicating an object depicted in an image and an action to be performed on that object (block 908). The at least one keyword may include a first keyword indicating the object and a second keyword indicating the action to be performed on the object. The messaging client application 104 may perform a scan of the image to identify multiple objects in the image, and detect objects from among the multiple objects based on the performed scan.

[0143] This action could correspond to providing object recognition information, causing the augmented reality content to provide object recognition information. In another example, the object could correspond to a problem to be solved, causing the action to correspond to solving the problem, and the augmented reality content to display the solution to the problem. In yet another example, the action could correspond to applying visual effects to objects depicted in an image, causing the augmented reality content to apply visual effects to objects depicted in an image.

[0144] The messaging client application 104 identifies the augmented reality content item corresponding to the action performed on the object from a plurality of augmented reality content items (block 910).

[0145] Identifying augmented reality content items may include sending a request to the messaging server system 112 to search for multiple augmented reality content items based on object attributes and action attributes, and receiving an instruction from the messaging server system 112 to receive augmented reality content items based on the request. The messaging server system 112 may be configured to search for multiple augmented reality content items by comparing object attributes and action attributes with predefined attributes associated with each of the multiple augmented reality content items.

[0146] The messaging client application 104 displays augmented reality content items (e.g., augmented reality content items) along with images (block 912). The messaging client application 104 can provide a carousel interface for display, which includes a corresponding icon for each of a plurality of augmented reality content items, and enables differentiated display of the icon for the augmented reality content item relative to the other icons within the carousel interface.

[0147] Figure 10 This is a block diagram 1000 illustrating a software architecture 1004 that can be installed on any or more devices described herein. The software architecture 1004 is supported by hardware such as a machine 1002 including a processor 1020, memory 1026, and I / O components 1038. In this example, the software architecture 1004 can be conceptualized as a stack of layers, where each layer provides specific functionality. The software architecture 1004 includes layers such as an operating system 1012, libraries 1010, frameworks 1008, and applications 1006. Operationally, application 1006 activates API calls 1050 via the software stack and receives messages 1052 in response to API calls 1050.

[0148] Operating system 1012 manages hardware resources and provides public services. Operating system 1012 includes, for example, a core 1014, services 1016, and drivers 1022. Core 1014 acts as an abstraction layer between the hardware layer and other software layers. For example, core 1014 provides memory management, processor management (e.g., scheduling), component management, network and security settings, and other functions. Services 1016 can provide other public services to other software layers. Driver 1022 is responsible for controlling or interfacing with the underlying hardware. For example, driver 1022 may include a display driver, a camera driver, a Bluetooth® or Bluetooth® Low Energy driver, a flash memory driver, a serial communication driver (e.g., a Universal Serial Bus (USB) driver), a Wi-Fi® driver, an audio driver, a power management driver, etc.

[0149] Library 1010 provides low-level public infrastructure used by application 1006. Library 1010 may include system library 1018 (e.g., the C standard library), which provides functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. Additionally, library 1010 may include API library 1024, such as media libraries (e.g., libraries for supporting the rendering and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Picture Experts Group (JPEG or JPG), or Portable Web Graphics (PNG)), graphics libraries (e.g., the OpenGL framework for rendering graphic content on a display in two-dimensional (2D) and three-dimensional (3D) formats), database libraries (e.g., SQLite, which provides various relational database functionalities), web libraries (e.g., WebKit, which provides web browsing functionality), etc. Library 1010 may also include a variety of other libraries 1028 to provide many other APIs to application 1006.

[0150] Framework 1008 provides high-level public infrastructure for use by application 1006. For example, framework 1008 provides various graphical user interface (GUI) functions, advanced resource management, and advanced location services. Framework 1008 can provide a wide range of other APIs that can be used by application 1006, some of which may be specific to a particular operating system or platform.

[0151] In an example implementation, application 1006 may include a home application 1036, a contacts application 1030, a browser application 1032, a book reader application 1034, a location application 1042, a media application 1044, a messaging application 1046 (e.g., corresponding to messaging client application 104), a game application 1048, and other applications in a broad category, such as third-party applications 1040. Application 1006 is a program that performs the functions defined in the program. One or more applications 1006 can be created using various programming languages, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In a particular example, third-party application 1040 (e.g., an application developed by an entity other than a platform-specific vendor using the Android™ or iOS™ Software Development Kit (SDK)) may be mobile software running on a mobile operating system such as iOS™, Android™, Windows® Phone, or another mobile operating system. In this example, a third-party application 1040 can activate API calls 1050 provided by the operating system 1012 to facilitate the functionality described herein.

[0152] Figure 11This is a graphical representation of machine 1100, in which instructions 1108 (e.g., software, programs, applications, applets, or other executable code) can be executed to cause machine 1100 to perform any or more of the methods discussed herein. For example, instructions 1108 can cause machine 1100 to perform any or more of the methods described herein. Instructions 1108 transform a generic, unprogrammed machine 1100 into a specific machine 1100 programmed to perform the described and illustrated functions in the manner described. Machine 1100 can operate as a standalone device or can be coupled (e.g., networked) to other machines. In a network deployment, machine 1100 can operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 1100 may include, but is not limited to, server computers, client computers, personal computers (PCs), tablet computers, laptop computers, netbooks, set-top boxes (STBs), PDAs, entertainment media systems, cellular phones, smartphones, mobile devices, wearable devices (e.g., smartwatches), smart home devices (e.g., smart appliances), other smart devices, web appliances, network routers, network switches, network bridges, or any machine capable of sequentially or otherwise executing instructions 1108 specifying actions to be taken by machine 1100. Furthermore, although only a single machine 1100 is shown, the term "machine" should also be understood to include a collection of machines that individually or jointly execute instructions 1108 to perform any one or more of the methods discussed herein.

[0153] Machine 1100 may include processor 1102, memory 1104, and I / O components 1144, which may be configured to communicate with each other via bus 1142. In an example implementation, processor 1102 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, processor 1106 and processor 1110 executing instruction 1108. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") capable of executing instructions simultaneously. Although Figure 11 Multiple processors 1102 are shown, but machine 1100 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0154] Memory 1104 includes main memory 1112, static memory 1114, and storage cells 1116, all accessible to processor 1102 via bus 1142. Main memory 1104, static memory 1114, and storage cells 1116 store instructions 1108 that implement any one or more of the methods or functions described herein. Instructions 1108 may also reside wholly or partially within main memory 1112, static memory 1114, machine-readable medium 1118 within storage cells 1116, at least one of processors 1102 (e.g., within the processor's cache memory), or any suitable combination thereof, during execution by machine 1100.

[0155] I / O component 1144 may include various components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurement results, etc. The specific I / O component 1144 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine is unlikely to include such a touch input device. It should be understood that I / O component 1144 may include... Figure 11 Many other components are not shown. In various example embodiments, I / O component 1144 may include output component 1128 and input component 1130. Output component 1128 may include visual components (e.g., displays such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tube (CRT) displays), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. Input component 1130 may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, photoelectric keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens or other haptic input components that provide position and / or force for touch or touch gestures), audio input components (e.g., microphones), optical sensor components (e.g., cameras), etc.

[0156] In another example implementation, I / O component 1144 may include biometric component 1132, motion component 1134, environmental component 1136 or positioning component 1138, and various other components. For example, biometric component 1132 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body posture, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), and identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition). Motion component 1134 includes accelerometer components (e.g., accelerometer), gravity sensor components, rotation sensor components (e.g., gyroscope), etc. Environmental component 1136 includes, for example, a lighting sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers for detecting ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones for detecting background noise), a proximity sensor component (e.g., an infrared sensor for detecting nearby objects), a gas sensor (e.g., a gas detection sensor for detecting the concentration of hazardous gases for safety reasons or for measuring pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. Positioning component 1138 includes a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer for detecting air pressure from which altitude can be obtained), an orientation sensor component (e.g., a magnetometer), etc.

[0157] Various technologies can be used to implement communication. I / O component 1144 also includes communication component 1140, which is operable to couple machine 1100 to network 1120 or device 1122 via coupler 1126 and coupler 1124, respectively. For example, communication component 1140 may include a network interface component or another suitable device to interface with network 1120. In further examples, communication component 1140 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Bluetooth® components (e.g., Bluetooth® Low Energy), Wi-Fi® components, and other communication components that provide communication via other forms. Device 1122 may be another machine or any peripheral device from a variety of peripheral devices (e.g., a peripheral device coupled via USB).

[0158] Furthermore, the communication component 1140 may detect identifiers or include components operable to detect identifiers. For example, the communication component 1140 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional barcodes, such as Universal Product Code (UPC) barcodes; multi-dimensional barcodes, such as Quick Response (QR) codes, Aztec codes, data matrices, dataglyphs, MaxiCodes, PDF417, hypercodes, UCC RSS-2D barcodes; and other optical codes) or an acoustic detection component (e.g., a microphone for identifying tagged audio signals). Additionally, various information can be derived via the communication component 1140, such as location via Internet Protocol (IP) geolocation, location via Wi-Fi® signal triangulation, location by detecting NFC beacon signals that can indicate a specific location, etc.

[0159] Various memories (e.g., memory 1104, main memory 1112, static memory 1114, and / or the memory of processor 1102) and / or storage units 1116 may store one or more sets of instructions and data structures (e.g., software) that embody or are used by any one or more of the methods or functions described herein. These instructions (e.g., instruction 1108), when executed by processor 1102, cause various operations to implement the disclosed embodiments.

[0160] Instructions 1108 can be sent or received over network 1120 using a transmission medium, via a network interface device (e.g., a network interface component included in communication component 1140), and using any of a number of known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instructions 1108 can be sent or received using a transmission medium via a coupling 1124 to device 1122 (e.g., a peer-to-peer coupling).

[0161] "Client device" refers to any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. Client devices can be, but are not limited to, mobile phones, desktop computers, laptop computers, portable digital assistants (PDAs), smartphones, tablet computers, ultrabooks, netbooks, multiple laptops, multiprocessor systems, microprocessor-based or programmable consumer electronics, game consoles, set-top boxes, or any other communication device that a user can use to access the network.

[0162] "Communications network" refers to one or more parts of a network, which can be an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), the Internet, a part of the Internet, a part of the Public Switched Telephone Network (PSTN), a Common Old-Style Telephone Service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, another type of network, or a combination of two or more such networks. For example, a network or part of a network may include a wireless network or a cellular network, and the coupling may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or other types of cellular or wireless coupling. In this example, coupling can implement any data transmission technology of various types, such as single-carrier radio transmission technology (1xRTT), evolved data optimization (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rate evolution of GSM (EDGE) technology, the 3rd Generation Partnership Project (3GPP) including 3G, fourth-generation wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Global Microwave Access Interoperability (WiMAX), Long Term Evolution (LTE) standards, other data transmission technologies defined by various standards setting organizations, other long-distance protocols, or other data transmission technologies.

[0163] A “component” refers to a device, physical entity, or logic having boundaries defined by function or subroutine calls, branch points, APIs, or other technologies provided for partitioning or modularizing specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processing. A component can be an encapsulated functional hardware unit designed for use with other components and can be part of a program that typically performs a specific function within a related function. Components can constitute software components (e.g., code embodied on a machine-readable medium) or hardware components. A “hardware component” is a tangible unit capable of performing certain operations and can be configured or arranged in some physical manner. In various example implementations, one or more computer systems (e.g., standalone computer systems, client computer systems, or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., an application or application portion) to perform certain operations as described herein. Hardware components can also be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware component may include dedicated circuitry or logic permanently configured to perform certain operations. Hardware components can be dedicated processors, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). Hardware components can also include programmable logic or circuitry that is temporarily configured by software to perform certain operations. For example, a hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific part of a machine) uniquely tailored to perform the configured function and is no longer a general-purpose processor. It will be understood that decisions to implement hardware components mechanically in dedicated and permanently configured circuitry or in temporarily configured (e.g., software-configured) circuitry may be driven by cost and time considerations. Therefore, the phrase "hardware component" (or "hardware-implemented component") should be understood to include tangible entities, i.e., entities that are physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain way or perform certain operations described herein. Considering the implementation where hardware components are temporarily configured (e.g., programmed), it is not necessary to configure or instantiate each hardware component in the hardware component at any given time. For example, in cases where the hardware components include a general-purpose processor configured as a dedicated processor via software, this general-purpose processor can be configured as different dedicated processors (e.g., including different hardware components) at different times. The software accordingly configures one or more specific processors to constitute a specific hardware component at one time and different hardware components at different times. Hardware components can provide information to and receive information from other hardware components. Accordingly, the described hardware components can be considered communicatively coupled.In the presence of multiple hardware components, communication can be achieved through signal transmission (e.g., via appropriate circuitry and buses) between or among the two or more hardware components. In embodiments where multiple hardware components are configured or instantiated at different times, such communication between hardware components can be achieved, for example, by storing information in a memory structure accessible to the multiple hardware components and retrieving information from that memory structure. For example, a hardware component can perform an operation and store the output of that operation in a communication-coupled memory device. Another hardware component can then access the memory device at a subsequent time to retrieve and process the stored output. Hardware components can also initiate communication with input or output devices and can operate on resources (e.g., information collection). The various operations of the example methods described herein can be performed, at least in part, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute processor-implemented components that operate to perform one or more operations or functions described herein. As used herein, "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the methods described herein can be implemented at least in part by processors, where a particular processor or one or more processors are examples of hardware. For example, at least some operations of the methods can be performed by one or more processors or processor-implemented components. Furthermore, one or more processors can also operate to support the execution of related operations in a “cloud computing” environment or as “Software as a Service” (SaaS). For example, at least some operations can be performed by a group of computers (as an example of a machine including processors), where these operations are accessible via a network (e.g., the Internet) and via one or more suitable interfaces (e.g., APIs). The execution of certain operations can be distributed among processors, residing not only within a single machine but also deployed across multiple machines. In some example implementations, the processor or processor-implemented component may reside in a single geographic location (e.g., within a home environment, office environment, or server cluster). In other example implementations, the processor or processor-implemented component may be distributed across multiple geographic locations.

[0164] "Computer-readable medium" refers to both machine storage media and transmission media. Therefore, these terms encompass both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" refer to the same thing and can be used interchangeably in this disclosure.

[0165] A "brief message" is a message that is accessible for a limited period of time. Brief messages can be text, images, videos, etc. The access time for a brief message can be set by the message sender. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting method, the message is temporary.

[0166] "Machine storage medium" refers to one or more storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions, routines, and / or data. Therefore, the above term should be considered to include, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium," "device storage medium," and "computer storage medium" refer to the same thing and are used interchangeably in this disclosure. The terms "machine storage medium," "computer storage medium," and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are covered by the term "signal medium."

[0167] A "processor" is any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and generates corresponding output signals applied to operate a machine. For example, a processor can be a Central Processing Unit (CPU), a Reduced Instruction Set Computing (RISC) processor, a Complex Instruction Set Computing (CISC) processor, a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), a Radio Frequency Integrated Circuit (RFIC), or any combination thereof. A processor can also be a multi-core processor with two or more independent processors (sometimes called "cores") that can execute instructions simultaneously.

[0168] "Signal medium" means any intangible medium capable of storing, encoding, or carrying instructions executable by a machine, and includes digital or analog communication signals or other intangible media that facilitate the communication of software or data. The term "signal medium" should be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal whose characteristics are set or altered in such a way as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same medium and may be used interchangeably in this disclosure.

[0169] Changes and modifications may be made to the disclosed embodiments without departing from the scope of this disclosure. Such and other changes or modifications are intended to be included within the scope of this disclosure and are set forth in the appended claims.

[0170] In addition, this disclosure can also be configured as follows.

[0171] (1). A method comprising:

[0172] The device's camera captures images via a messaging application running on the device;

[0173] The messaging application receives voice input to select augmented reality content to be displayed along with the image;

[0174] Identify at least one keyword included in the voice input;

[0175] Determine the at least one keyword that indicates an object depicted in the image and an action performed on the object;

[0176] Identify, from a plurality of augmented reality content items, the augmented reality content item corresponding to the action performed on the object; and

[0177] The augmented reality content item is displayed together with the image.

[0178] (2). The method according to (1) further includes:

[0179] Perform a scan of the image to identify multiple objects in the image; and

[0180] Based on the execution of the scan, the object is detected from the plurality of objects.

[0181] (3). According to the method of (1), determining the at least one keyword includes:

[0182] Send a request to the speech recognition service to perform speech recognition based on the speech input; and

[0183] Based on sending the request, the at least one keyword is received from the speech recognition service.

[0184] (4). According to the method of (3), wherein the first part of the voice input includes a trigger word, and

[0185] Wherein, the at least one keyword is based on a second part of the voice input that does not include the trigger word.

[0186] (5). According to the method of (1), wherein identifying the augmented reality content item includes:

[0187] Send a request to the server to search for the plurality of augmented reality content items based on the attributes of the object and the attributes of the action; and

[0188] Instructions to receive the augmented reality content item from the server based on sending the request.

[0189] (6). The method according to (5), wherein the server is configured to search the plurality of augmented reality content items by comparing the attributes of the object and the attributes of the action with predefined attributes associated with each of the plurality of augmented reality content items.

[0190] (7). The method according to (1) further includes:

[0191] A carousel interface is provided for display, the carousel interface including a corresponding icon for each of the plurality of augmented reality content items; and

[0192] This allows for a differentiated display of the icon for the augmented reality content item relative to the other icons within the carousel interface.

[0193] (8). According to the method of (1), wherein the action corresponds to providing identification information of the object, and

[0194] The augmented reality content provides identification information for the object.

[0195] (9). According to the method described in (1), wherein the object corresponds to the problem to be solved,

[0196] Wherein, the action corresponds to solving the problem, and

[0197] The augmented reality content displays the solution to the problem.

[0198] (10). The method according to (1), wherein the action corresponds to applying a visual effect to the object depicted in the image, and

[0199] The augmented reality content enables the application of the visual effects to the objects depicted in the image.

[0200] (11). According to the method of (1), wherein the at least one keyword includes a first keyword indicating the object and a second keyword indicating the action to be performed on the object.

[0201] (12). An apparatus comprising:

[0202] Processor; and

[0203] A memory for storing instructions that, when executed by the processor, cause the processor to:

[0204] The device's camera captures images via a messaging application running on the device;

[0205] The messaging application receives voice input to select augmented reality content to be displayed along with the image;

[0206] Identify at least one keyword included in the voice input;

[0207] Determine the at least one keyword that indicates an object depicted in the image and an action performed on the object;

[0208] Identify, from a plurality of augmented reality content items, the augmented reality content item corresponding to the action performed on the object; and

[0209] The augmented reality content item is displayed together with the image.

[0210] (13). The device according to (12), wherein the instructions further cause the processor to:

[0211] Perform a scan of the image to identify multiple objects in the image; and

[0212] Based on the execution of the scan, the object is detected from the plurality of objects.

[0213] (14). According to the device described in (12), determining the at least one keyword includes:

[0214] Send a request to the speech recognition service to perform speech recognition based on the speech input; and

[0215] Based on sending the request, the at least one keyword is received from the speech recognition service.

[0216] (15). The device according to (14), wherein the first part of the voice input includes a trigger word, and

[0217] Wherein, the at least one keyword is based on a second part of the voice input that does not include the trigger word.

[0218] (16). According to the device of (12), wherein identifying the augmented reality content item includes:

[0219] Send a request to the server to search for the plurality of augmented reality content items based on the attributes of the object and the attributes of the action; and

[0220] Instructions to receive the augmented reality content item from the server based on sending the request.

[0221] (17). The device according to (16), wherein the server is configured to search the plurality of augmented reality content items by comparing the attributes of the object and the attributes of the action with predefined attributes associated with each of the plurality of augmented reality content items.

[0222] (18). The device according to (12), wherein the instructions further cause the processor to:

[0223] A carousel interface is provided for display, the carousel interface including a corresponding icon for each of the plurality of augmented reality content items; and

[0224] This allows for a differentiated display of the icon for the augmented reality content item relative to the other icons within the carousel interface.

[0225] (19). The device according to (12), wherein the action corresponds to providing identification information of the object, and

[0226] The augmented reality content provides identification information for the object.

[0227] (20). A non-transitory computer-readable storage medium, the computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to:

[0228] The device's camera captures images via a messaging application running on the device;

[0229] The messaging application receives voice input to select augmented reality content to be displayed along with the image;

[0230] Identify at least one keyword included in the voice input;

[0231] Determine the at least one keyword that indicates an object depicted in the image and an action performed on the object;

[0232] Identify, from a plurality of augmented reality content items, the augmented reality content item corresponding to the action performed on the object; and

[0233] The augmented reality content item is displayed together with the image.

Claims

1. A method for displaying augmented reality content, comprising: Receive voice input to select augmented reality content to be displayed along with the image; Identify at least one keyword included in the voice input; The at least one keyword is determined to indicate the action to be performed on the image; Determine the first attribute of the object depicted in the image; Assign weights to each of the first attributes of the object; The multiple augmented reality content items are ranked based on the assigned weights and the second attribute of the action; Based on the ranking, select the augmented reality content item with the highest ranking from the plurality of augmented reality content items; as well as Activate the highest-ranking augmented reality content item for the image.

2. The method according to claim 1, further comprising: Perform a scan of the image to identify multiple objects in the image; as well as Based on the execution of the scan, objects are detected from the plurality of objects.

3. The method according to claim 1, wherein, Determining the at least one keyword includes: Send a request to the speech recognition service to perform speech recognition based on the speech input; and Based on sending the request, the at least one keyword is received from the speech recognition service.

4. The method according to claim 3, wherein, The first part of the voice input includes a trigger word, and Wherein, the at least one keyword is based on a second part of the voice input that does not include the trigger word.

5. The method according to claim 1, wherein, The at least one keyword includes a first keyword indicating the object and a second keyword indicating the action to be performed on the object.

6. The method according to claim 1, further comprising: An interface with user-selectable elements is displayed in a ranking order based on the ranking, for activating the remaining items among the plurality of augmented reality content items.

7. The method according to claim 6, in, The interface is a carousel interface, which has a corresponding user-selectable icon for each of the plurality of augmented reality content items, and The carousel interface displays the icon of the augmented reality content item differently from the other icons.

8. An apparatus comprising: At least one processor; as well as A memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations, the operations including: Receive voice input to select augmented reality content to be displayed along with the image; Identify at least one keyword included in the voice input; The at least one keyword is determined to indicate the action to be performed on the image; Determine the first attribute of the object depicted in the image; Assign weights to each of the first attributes of the object; The multiple augmented reality content items are ranked based on the assigned weights and the second attribute of the action; Based on the ranking, select the augmented reality content item with the highest ranking from the plurality of augmented reality content items; and Activate the highest-ranking augmented reality content item for the image.

9. The device according to claim 8, wherein the operation further comprises: Perform a scan of the image to identify multiple objects in the image; as well as Based on the execution of the scan, objects are detected from the plurality of objects.

10. The device according to claim 8, wherein, Determining the at least one keyword includes: Send a request to the speech recognition service to perform speech recognition based on the speech input; and Based on sending the request, the at least one keyword is received from the speech recognition service.

11. The device according to claim 10, wherein, The first part of the voice input includes a trigger word, and Wherein, the at least one keyword is based on a second part of the voice input that does not include the trigger word.

12. The device according to claim 8, wherein, The at least one keyword includes a first keyword indicating the object and a second keyword indicating the action to be performed on the object.

13. The device according to claim 8, wherein the operation further comprises: An interface with user-selectable elements is displayed in a ranking order based on the ranking, for activating the remaining items among the plurality of augmented reality content items.

14. The device according to claim 13, in, The interface is a carousel interface, which has a corresponding user-selectable icon for each of the plurality of augmented reality content items, and The carousel interface displays the icon of the augmented reality content item differently from the other icons.

15. A non-transitory computer-readable storage medium, the computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform an operation, the operation comprising: Receive voice input to select augmented reality content to be displayed along with the image; Identify at least one keyword included in the voice input; The at least one keyword is determined to indicate the action to be performed on the image; Determine the first attribute of the object depicted in the image; Assign weights to each of the first attributes of the object; The multiple augmented reality content items are ranked based on the assigned weights and the second attribute of the action; Based on the ranking, select the augmented reality content item with the highest ranking from the plurality of augmented reality content items; as well as Activate the highest-ranking augmented reality content item for the image.

16. The non-transitory computer-readable storage medium according to claim 15, further comprising: Perform a scan of the image to identify multiple objects within the image; and Based on the execution of the scan, objects are detected from the plurality of objects.

17. The non-transitory computer-readable storage medium according to claim 15, wherein, Determining the at least one keyword includes: Send a request to the speech recognition service to perform speech recognition based on the speech input; and Based on sending the request, the at least one keyword is received from the speech recognition service.

18. The non-transitory computer-readable storage medium according to claim 17, wherein, The first part of the voice input includes a trigger word, and Wherein, the at least one keyword is based on a second part of the voice input that does not include the trigger word.

19. The non-transitory computer-readable storage medium according to claim 15, wherein, The at least one keyword includes a first keyword indicating the object and a second keyword indicating the action to be performed on the object.

20. The non-transitory computer-readable storage medium of claim 15, further comprising: An interface with user-selectable elements is displayed in a ranking order based on the ranking, for activating the remaining items among the plurality of augmented reality content items.

Images