Method and system for enhancing video call experience and machine readable storage medium

Abstract

Methods, systems, and machine-readable storage media for enhancing video call experience are provided. The method includes presenting a first user-selectable element at a first device, the first user-selectable element operable to inhibit video in a video call session between the first device and a second device; in response to detecting activation of the first user-selectable element, presenting, at the first device, a second user-selectable element operable to activate avatar communication processing; and in response to detecting the activation of the second user-selectable element, activating an avatar communication process for the first device, the avatar communication process comprising: generating an avatar animation video based on avatar animation data, the avatar animation data includes a plurality of data points representing a position of a facial feature in an area representing a face; and causing the avatar animation video to be displayed on a display.

Description

[0001] This application is a divisional application of the PCT application with international application number PCT / US2022 / 081732 filed on December 16, 2022 and invention name “Avatar Call Platform”. The date on which the PCT application entered the Chinese national phase is June 18, 2024, the national application number is 202280083830.8, and the priority date is December 21, 2021.

[0002] Priority application

[0003] This application is a continuation application of U.S. patent application No. 17 / 558,254 filed on December 21, 2021, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0004] The present disclosure generally relates to facilitating interaction between client devices over a network. Background Art

[0005] The popularity of computer-implemented tools that allow users to access content online and interact with other users continues to grow. For example, there are various computer-implemented tools that allow users to interact and share content with other users through messaging applications. Some of these computer-implemented tools (referred to as applications or apps) can be designed to run on mobile devices such as phones, tablets, or watches. Existing messaging applications allow users to exchange text messages, as well as make and receive phone calls. Summary of the invention

[0006] The present invention provides a method for enhancing a video call experience, comprising: presenting a first user-selectable element at a first device, the first user-selectable element being operable to disable video in a video call session between the first device and a second device; in response to detecting activation of the first user-selectable element, presenting a second user-selectable element at the first device being operable to activate an avatar communication process; and in response to detecting activation of the second user-selectable element, activating the avatar communication process for the first device, the avatar communication process comprising: generating an avatar animation video based on avatar animation data, the avatar animation data comprising a plurality of data points representing positions of facial features in an area representing a face; and causing the avatar animation video to be displayed on a display.

[0007] The present invention provides a system for enhancing video call experience, comprising: one or more processors; and a non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium comprising instructions, which when executed by the one or more processors cause the system to perform operations, the operations comprising: presenting a first user-selectable element at a first device, the first user-selectable element being operable to disable video in a video call session between the first device and a second device; in response to detecting activation of the first user-selectable element, presenting a second user-selectable element at the first device that is operable to activate an avatar communication processing; and in response to detecting activation of the second user-selectable element, activating the avatar communication processing for the first device, the avatar communication processing comprising: generating an avatar animation video based on avatar animation data, the avatar animation data comprising multiple data points representing positions of facial features in an area representing a face; and causing the avatar animation video to be displayed on a display.

[0008] The present invention provides a non-transitory machine-readable storage medium having instruction data executable by a machine so that the machine performs operations, the operations comprising: presenting a first user-selectable element at a first device, the first user-selectable element being operable to disable video in a video call session between the first device and a second device; in response to detecting activation of the first user-selectable element, presenting a second user-selectable element at the first device being operable to activate avatar communication processing; and in response to detecting activation of the second user-selectable element, activating avatar communication processing for the first device, the avatar communication processing comprising: generating an avatar animation video based on avatar animation data, the avatar animation data comprising multiple data points representing positions of facial features in an area representing a face; and causing the avatar animation video to be displayed on a display. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In the accompanying drawings, like reference numerals may describe similar components in different views, and the drawings are not necessarily drawn to scale. To easily identify the discussion of any particular element or action, the highest digit or digits in the reference numeral refer to the figure number in which the element is first introduced. Some examples are shown by way of example and not limitation in the figures of the accompanying drawings, in which:

[0010] Figure 1 is a diagrammatic representation of a networked environment in which an avatar calling platform may be deployed, according to some examples.

[0011] Figure 2 is a diagrammatic representation of a messaging system having both client-side and server-side functionality and including an avatar calling platform, according to some examples.

[0012] Figure 3is a diagrammatic representation of data structures as maintained in a database according to some examples.

[0013] Figure 4 is a flow diagram illustrating an avatar communication process during a video call session between a first client device and a second client device according to some examples.

[0014] Figure 5 A user interface displayed during a video call session between a first client device and a second client device when an avatar communication process is activated is shown according to some examples.

[0015] Figure 6 is a graphical representation of the training and use of a machine learning program based on some examples.

[0016] Figure 7 is a diagrammatic representation of a machine in the form of a computer system according to some examples within which a set of instructions may be executed for causing the machine to perform any one or more of the methodologies discussed herein. DETAILED DESCRIPTION

[0017] Examples of the present disclosure improve the functionality of electronic software and systems by enhancing the user experience of applications that allow real-time reception and transmission of audio and video signals between client devices. Users can talk to each other during a call session while seeing each other on the screens of their respective devices, a process commonly referred to as a video call. A video calling system provides video calling functionality through a video calling application executed on a client device, wherein the video calling application executed on the client device is supported by an associated backend system.

[0018] The video calling application may be configured to provide the user with the option to allow video, in which case the other user who is a party to the video call can see the user on their screen, or to provide the user with the option to disallow or turn off the video feed, in which case the other user's screen displays a still image or just a solid color background in the area designated for the live video feed. In some cases, allowing the video call may not be desirable, such as when the person who is a party to the call feels that they do not look their best. Disabling the video feed is not always a satisfactory experience, especially when the other party to the call allows the video feed to be displayed on the other party's screen.

[0019] The technical problem of generating a video feed that represents a user who is a party to a video call in a manner that activates the caller's visual presence without transmitting the user's real-time video is solved by configuring the video calling system to include an avatar calling platform.

[0020] The avatar calling platform is configured to generate and display a video of an animated figure representing the caller during the call, as the face of the animated figure moves in a manner that matches what the caller says during the call. The video of the animated figure representing the caller during the call is called an avatar animation video. The animated figure representing the user during the call is called the user's avatar image, avatar representation, or simply the user's avatar. The avatar can be selected by the user or automatically selected by the avatar calling platform. In some examples, the avatar somewhat resembles the user (e.g., in terms of hair color and length, presence or absence of glasses and / or facial hair, etc.), or it can be any animated character, such as an animated puppy or a character from a children's cartoon.

[0021] The avatar calling platform provides an intermediate option between voice calling without video and video calling in that it conveys a visual indication of the real-time facial movements of the person speaking while providing a degree of privacy by shielding the person who selects the avatar calling option from being seen in real-time (e.g., when they have just woken up). This intermediate option between voice calling without video and video calling, in which an area of ​​the user's screen designated for a real-time video feed displays an animated graphic representing the caller during the call, is referred to as the avatar calling option for purposes of this description. The following describes the processing performed when a video calling application is executed on a client device when the avatar communication process is activated, which is referred to as the avatar communication process.

[0022] When a call session between a first client device and a second client device is ongoing, a video call application executed on the first client device continuously receives call data from relevant video and / or audio sensors local to the first client device and sends it to the second client device via a network. For the purposes of this description, call data originating locally from the video and / or audio sensors of a client device is referred to as local call data about the client device. Based on the local call data, the avatar call platform generates avatar animation data, which can be used together with an avatar image to create an animated character that conveys, for example, facial expressions and / or lip movements that match what the caller said. For the purposes of this description, an animated character that conveys facial expressions and / or lip movements that match what the caller said is referred to as an avatar animation video. In some examples, the local call data used to generate the avatar animation data can be a video feed obtained by means of a digital image sensor of a camera device of the client device. In other examples, the local call data used to generate the avatar animation data is audio data obtained through a microphone of the client device.

[0023] In some examples, the avatar animation data in the form of hints generated for each frame of the video represents the facial expression corresponding to the video frame. Hints refer to data points that describe the corresponding locations of facial features in the area representing the face. For example, to draw a smile, as described below Figure 5 As shown in area 510 in , hints associated with corresponding positions of several points in the representation of the mouth can be generated to indicate that the left side of the mouth should be 5 points above the neutral position, the middle should be 1 point above the neutral position, and the right side should be 4 points above the neutral position.

[0024] The avatar animation data is used to generate an avatar animation video, which is displayed on the local screen of the client device so that the user can see the result of the avatar call option being activated. The avatar call platform is also configured to transmit information for displaying the avatar animation video on the screens of other client devices participating in the call session. In some examples, a compressed version of the avatar animation video generated at the first client device is transmitted to the second client device for display on the screen of the other client device. In other examples, the avatar call platform is configured to transmit the avatar animation data (which may be compressed first) to the second client device, and the avatar animation video is generated at the second client device and then displayed on the screen of the other client device.

[0025] A user interacting with the video calling application can activate an avatar communication process in the video calling application, for example, when the user initiates or accepts a call or at any time during the call. In some examples, the avatar calling platform can detect a user requesting to disable the capture of live video by their client device during the call, and in response, display a user interface (UI) including a user-selectable element that can be activated to participate in the avatar calling platform. In some examples, the avatar calling platform displays a UI including a user-selectable element that can be activated to participate in the avatar calling platform while the camera is still on.

[0026] In some examples, the avatar calling platform is used in the context of a messaging system that hosts the backend services of an associated messaging client. The messaging system may be referred to as a video calling system, and the messaging client may be referred to as a video calling application. Figures 1 to 3 The messaging system is further described. Although the avatar calling platform is described below in the context of a messaging system, the methods described herein may be advantageously used in a variety of computer-implemented applications that provide video calling functionality.

[0027] Networked computing environment

[0028] Figure 11 is a block diagram illustrating an example messaging system 100 for exchanging data (e.g., messages and associated content) over a network. The messaging system 100 includes multiple instances of client devices 102, each of the multiple instances hosting a number of applications including a messaging client 104. Each messaging client 104 is communicatively coupled to other instances of the messaging client 104 and a messaging server system 108 via a network 106 (e.g., the Internet).

[0029] The message client 104 is able to communicate and exchange data with another message client 104 and a message server system 108 via the network 106. The data exchanged between the message clients 104 and between the message clients 104 and the message server system 108 includes functions (e.g., commands to activate functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0030] The message server system 108 provides server-side functionality to a particular message client 104 via the network 106. Although certain functions of the message system 100 are described herein as being performed by the message client 104 or by the message server system 108, the location of certain functions in the message client 104 or within the message server system 108 may be a design choice. For example, it may be technically preferable to initially deploy certain technologies and functions within the message server system 108, but later migrate the technologies and functions to the message client 104 where the client device 102 has sufficient processing power.

[0031] The message server system 108 supports various services and operations provided to the message client 104. Such operations include sending data to the message client 104, receiving data from the message client 104, and processing data generated by the message client 104. As an example, the data may include message content, client device information, geographic location information, media enhancement and overlay, message content persistence conditions, social network information, live event information, and images and videos captured by the front camera of the associated client device using customized image reprocessing. The data exchange within the message system 100 is activated and controlled by functions available via the user interface (UI) of the message client 104. The message client 104 presents a camera view UI that displays the output of the digital image sensor of the camera provided by the client device 102, and also displays various user selectable elements that can be activated by touching the area of ​​the screen displaying the user selectable elements. In some examples, the message client 104 is configured to include or access the avatar call platform described herein.

[0032] Turning now specifically to the message server system 108, an application program interface (API) server 110 is coupled to and provides a programming interface to an application server 112. The application server 112 is communicatively coupled to a database server 118, which facilitates access to a database 120 that stores data associated with messages processed by the application server 112. Similarly, a web server 124 is coupled to the application server 112 and provides a web-based interface to the application server 112. To this end, the web server 124 processes incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0033] The application program interface (API) server 110 receives and sends message data (e.g., commands and message payloads) between the client device 102 and the application server 112. Specifically, the application program interface (API) server 110 provides a set of interfaces (e.g., routines and protocols) that the message client 104 can call or query to activate the functions of the application server 112. The application program interface (API) server 110 exposes various functions supported by the application server 112, including: account registration; login function; sending messages from a particular message client 104 to another message client 104 via the application server 112; sending media files (e.g., images or videos) from the message client 104 to the message server 114 and for possible access by another message client 104; setting of media data collections (e.g., stories); retrieving a friend list of the user of the client device 102; retrieving such a collection; retrieving messages and content; adding and removing entities (e.g., friends) in an entity graph (e.g., a social graph); locating friends in a social graph; and opening application events (e.g., related to the message client 104).

[0034] The application server 112 hosts several server applications and subsystems, including, for example, a message server 114, a content processing server 116, and a social network server 122. The message server 114 implements several message processing technologies and functions, particularly those related to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the message client 104. As will be described in further detail, text and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or galleries). These collections are then made available to the message client 104. Given the hardware requirements for other processor and memory intensive processing of data, such processing can also be performed on the server side by the message server 114.

[0035] The application server 112 also includes a content processing server 116 that is dedicated to performing various processing operations on content such as images and videos. Information generated by the various processing operations on the content can be transmitted between client devices executing corresponding message clients 104 via the message server 114. In some examples, the avatar calling platform is supported on the server side by the content processing server 116 and the message server 114.

[0036] The social network server 122 supports various social networking functions and services and makes them available to the message server 114. To this end, the social network server 122 maintains and accesses the entity graph 306 (e.g., Figure 3 Examples of functions and services supported by social network server 122 include identifying other users in messaging system 100 that have a "friend" relationship with a particular user or that the particular user is "following," and identifying interests and other entities of a particular user.

[0037] System Architecture

[0038] Figure 2 1 is a block diagram showing additional details about the messaging system 100 according to some examples. Specifically, the messaging system 100 is shown to include a messaging client 104 and an application server 112. The messaging system 100 contains several subsystems that are supported on the client side by the messaging client 104 and on the server side by the application server 112. These subsystems include, for example, an enhancement system 202, an animation delivery system 206, and an avatar animation generator 204.

[0039] The enhancement system 202 provides various functions to enhance (e.g., annotate or otherwise modify or edit) media content. For example, the enhancement system 202 provides functions related to the generation and publication of media for communications facilitated by the messaging system 100. Media including avatar images can be stored in the database 120 and accessed through the database server 118.

[0040] In some examples, the augmentation system 202 includes or provides access to an AR component, which can be implemented using a programming language suitable for application development, such as JavaScript or Java, and is identified in the message server system by a corresponding AR component identifier. The AR component may include or reference various image processing operations corresponding to image modification, filtering, media overlay, transformation, etc. These image processing operations can provide an interactive experience of a real-world environment, in which objects, surfaces, backgrounds, lighting, etc. captured by a digital image sensor or camera are enhanced by computer-generated perceptual information. In this case, the AR component includes a collection of data, parameters, and other assets required to apply a selected augmented reality experience to an image or video feed.

[0041] In some examples, the AR component includes a module configured to modify or transform image data presented within a graphical user interface (GUI) of a client device in some manner. For example, the AR component data can be used to perform complex additions or transformations to content images, such as adding bunny ears to a person's head in a video clip, adding a floating heart with a background color to a video clip, changing the scale of a person's features within a video clip, generating an avatar animation video, or many other such transformations. This includes both real-time modifications, such as generating an avatar animation video during avatar communication processing, and modifications to stored content, such as video clips in a gallery that can be modified using the AR component.

[0042] Various augmented reality functions that can be provided by the AR component include detecting objects (e.g., faces, hands, bodies, cats, dogs, surfaces, objects, etc.) in video frames, tracking these objects as they leave, enter, and move around the field of view, and modifying or transforming them as they are tracked. In various examples, different methods for implementing such transformations can be used. For example, some examples may involve generating a 3D mesh model of one or more objects, and using transformations and animated textures of the model within the video to implement the transformations. In other examples, tracking of points on an object can be used to place an image or texture at the tracked location, which image or texture can be two-dimensional or three-dimensional. In another example, neural network analysis of video frames can be used to place images, models, or textures in content (e.g., frames of an image or video). AR component data refers to both images, models, and textures used to create transformations in content, as well as additional modeling and analysis information required to implement such transformations using object detection, tracking, and placement.

[0043] In some examples, the augmentation system 202 is utilized by or is part of the avatar calling platform, in that by using the augmentation system 202 to engage in the avatar calling option, the augmentation system 202 can track objects (e.g., body objects, facial objects, etc.) in the video and replace the face or entire body detected by the digital image sensor (original camera) with an animated avatar video. The augmentation system 202 is configured to analyze local call data (video feed and / or audio) and generate information representing various facial expressions of the face depicted in the frames of the video feed based on the local call data. In the context of the avatar calling platform, the information representing the facial expressions and the movement of facial features is referred to as avatar animation data.

[0044] The augmentation system 202 includes or is in communication with an avatar animation generator 204. The avatar animation generator 204 is configured to generate an avatar animation video based on the avatar animation data and the avatar image.

[0045] The animation delivery system 206 is configured to transmit information required to render the avatar animation video generated by the avatar animation generator 204 from one client device that is a party to the video call to another client device that is a party to the video call. Such information can be a compressed version of the avatar animation video. In some examples, the avatar call platform is configured so that the animation delivery system 206 transmits the avatar animation data or a compressed version of the avatar animation data from the first client device to the second client device, and the avatar animation generator 204 generates the avatar animation video based on the avatar animation data at the second client device. It will be noted that for the purposes of this description, the various components involved in the avatar communication process, such as the enhancement system 202, the avatar animation generator 204, and the animation delivery system 206, are collectively referred to as the avatar call platform.

[0046] Data Architecture

[0047] Figure 3 is a schematic diagram illustrating a data structure 300 that may be stored in database 120 of message server system 108 according to some examples. Although the contents of database 120 are shown as including several tables, it should be understood that data may be stored in other types of data structures (e.g., as an object-oriented database).

[0048] The database 120 includes message data stored in the message table 302. For any particular message, the message data includes at least message sender data, message recipient (or receiver) data, and payload. The payload of the message may include content generated using custom image reprocessing. Figure 4 Additional details regarding information that may be included in a message and included within the message data stored in message table 302 are described.

[0049] The entity table 304 stores entity data and is linked (e.g., referenced) to the entity graph 306 and profile data 308. The entities for which records are maintained within the entity table 304 may include individuals, corporate entities, organizations, objects, places, events, etc. Regardless of the entity type, any entity about which the message server system 108 stores data may be an identified entity. Each entity is provided with a unique identifier as well as an entity type identifier (not shown).

[0050] The entity graph 306 stores information about relationships and associations between entities. By way of example only, such relationships may be social, professional (e.g., working at a common company or organization), interest-based, or activity-based. With reference to functionality provided by the AR component, the entity graph 306 stores the following information: in the event that the AR component is configured to allow the use of a portrait image of a user other than a portrait image of the user controlling the associated client device to modify a target media content object, this information may be used to determine an additional profile connected to the profile of the user representing the associated client device being controlled. As mentioned above, the portrait image of the user may be stored in a user profile representing the user in the messaging system.

[0051] Profile data 308 stores multiple types of profile data about a particular entity. Based on the privacy settings specified by the particular entity, profile data 308 can be selectively used and presented to other users of the messaging system 100. In the case where the entity is a person, profile data 308 includes, for example, a user name, phone number, address, settings (e.g., notification and privacy settings), and an avatar representation (or a collection of such avatar representations) selected by the user or automatically assigned. Such an avatar representation can be referred to as an avatar image or a simple avatar. A particular user can then selectively include one or more of these avatar representations within the content of a message transmitted via the messaging system 100 and on a map interface displayed to other users by the messaging client 104. A collection of avatar representations can include a "status avatar" that presents a graphical representation of a status or activity that a user can choose to convey at a particular time. In some examples, an animated version of a user's avatar is used in a video call, replacing the caller's live video feed with an animated avatar video, wherein the face of the animated character moves in a manner that matches what the caller says during the call.

[0052] The database 120 also stores enhancement data in an enhancement table 310. The enhancement data is associated with and applied to videos (whose data is stored in a video table 314) and images (whose data is stored in an image table 316). In some examples, the enhancement data is used by various AR components. An example of enhancement data is an augmented reality (AR) tool that can be used in an AR component to implement image transformation. Image transformation includes real-time modification of an image (e.g., a video frame) as it is captured using a digital image sensor of the client device 102. The modified image is displayed on the screen of the client device 102 together with the modification.

[0053] AR tools can also be used to modify stored content, such as modifying video clips or still images stored in a gallery. In a client device 102 that has access to multiple AR tools, a user can apply different AR tools (e.g., by engaging different AR components configured to utilize different AR tools) to a single video clip to see how different AR tools will modify the same video clip. For example, by selecting different AR tools for the same captured content, multiple AR tools that apply different pseudo-random movement models can be applied to the same captured content. Similarly, real-time video capture can be used with the modifications shown to show how the video image currently captured by the digital image sensor of the camera provided by the client device 102 will modify the captured data. For example, an avatar animation video can be generated based on the real-time video captured by the digital image sensor of the client device 102 and presented in place of the real-time video during a video call session between client devices. Such data can be displayed only on the screen without being stored in the memory, or the content captured by the digital image sensor can be recorded and stored in the memory with or without modification (or in both cases). The messaging client 104 may be configured to include a preview feature that can simultaneously display in different windows of the display what modifications produced by different AR tools will look like. For example, this can allow a user to simultaneously view multiple windows with different pseudo-random animations presented on the display.

[0054] In some examples, when a specific modification and content to be converted are selected, the computing device identifies the elements to be converted, and then detects and tracks whether they exist in the video frame. The object elements are modified according to the modification request, thereby converting the frame of the video stream. The frame conversion of the video stream can be performed in different ways for different types of conversions. For example, for frame conversions that mainly involve changes in the form of object elements, feature points of each element of the object are calculated (for example, using active shape models (ASM) or other known methods). Then, a grid based on feature points is generated for each of at least one element of the object. This grid is used to track the next stage of the object element in the video stream. During the tracking process, the above-mentioned grid of each element is aligned with the position of each element. Then, other points are generated on the grid. A first group of first points is generated for each element based on the modification request, and a group of second points is generated for each element based on the group of first points and the modification request. Then, the frame of the video stream can be transformed by modifying the object element based on the group of first points and the group of second points and the grid. In this way, the background of the modified object can also be changed or distorted by tracking and modifying the background.

[0055] In some examples, a transformation that changes certain areas of an object using elements of an object can be performed by calculating feature points of each element of the object and generating a grid based on the calculated feature points. Points are generated on the grid, and then various areas are generated based on the points. The elements of the object are then tracked by aligning the area of ​​each element with the position of each element in at least one element, and the properties of the area can be modified based on the modification request, thereby transforming the frame of the video stream. Depending on the specific modification requirements, the properties of the above-mentioned areas can be transformed in different ways. Such modifications can include: changing the color of the area; removing at least part of the area from the frame of the video stream; including one or more new objects in the area based on the modification request; and modifying or distorting the elements of the area or object. In various examples, any combination of such modifications or other similar modifications can be used. For certain models to be animated, some feature points can be selected as control points to determine the entire state space of model animation options.

[0056] As mentioned above, the video table 314 stores video data, which in some examples is associated with a message whose record is maintained within the message table 302. Similarly, the image table 316 stores image data associated with a message whose message data is stored in the entity table 304. The entity table 304 may associate various enhancements from the enhancement table 310 with various images and videos stored in the image table 316 and the video table 314.

[0057] Process flow and user interface

[0058] Figure 4is a flow chart of a method 400 for enhancing a user's video call experience according to some examples. The method 400 may be performed by processing logic that may include hardware (eg, dedicated logic, programmable logic, microcode, etc.), software, or a combination of both.

[0059] In some examples, some or all of the processing logic resides in Figure 1 The client device 102 and / or Figure 1 The method 400 begins at operation 410, where the avatar calling platform activates an avatar communication process at a first client device executing a video calling application. When the avatar communication process is activated and a call session between the first client device and the second client device is ongoing, the video calling application continuously receives local call data from relevant video and / or audio sensors local to the first client device and sends it to the second client device over the network.

[0060] The avatar communication process can be activated in response to detecting a request associated with a user-selectable element presented on a user interface (UI) generated by the messaging system and displayed by the video call application at the first client device. In some examples, in response to detecting a request to suppress video in the video call session, a user-selectable element operable to activate the avatar communication process is presented.

[0061] At operation 420, the avatar call platform generates avatar animation data based on the local call data. As described above, the avatar animation data can be used to generate an animated character that conveys, for example, facial expressions and / or lip movements that match what the caller says. In some examples, the local call data used to generate the avatar animation data can be a video feed obtained by means of a digital image sensor of a camera device of a client device. For example, the avatar call platform can use an AR component to detect a facial object in a video, and generate avatar animation data based on the features of the facial object in the video. In another example, the avatar call platform can use an AR component to detect a hand object in a video, and generate avatar animation data based on the features of the hand object in the video, which can cause the avatar animation data to represent, for example, a raised hand. In some examples, the local call data used to generate the avatar animation data is audio data obtained by a microphone of a client device. In some examples, the avatar animation data is a prompt form representing a facial expression. The avatar animation data can be generated using a machine learning model (e.g., a trained neural network) provided with the augmentation system 202. Figure 6 A graphical representation of the training and use of a machine learning program is shown, which will be further described below.

[0062] At operation 430, the avatar animation generator 204 generates an avatar animation video including, for example, an animated character conveying facial expressions and / or lip movements that match what the caller is saying. The avatar animation video is generated based on avatar animation data (such as, for example, prompts) and an avatar image. In some examples, the video call session is associated with a user profile representing a user in the messaging system, and the avatar image is a cartoon representation of the user in the messaging system.

[0063] At operation 440, the avatar call platform causes the avatar animation video to be displayed on a display device, such as, for example, a screen of a first client device, so that the user can see the results of the avatar communication process. The avatar call platform is also configured to transmit information for causing the avatar animation video to be displayed on the screens of other client devices participating in the call session. In some examples, the information for causing the avatar animation video to be displayed on the screens of other client devices is a compressed version of the avatar animation video generated at the first client device. In some examples, the information for causing the avatar animation video to be displayed on the screens of other client devices is avatar animation data (which may be compressed first), in which case the avatar animation video is generated at the second client device and then displayed on the screen of the second client device.

[0064] Figure 5 A UI 500 is shown displayed during a video call session between a first client device and a second client device when an avatar communication process is activated according to some examples. The UI 500 shown as presented on a screen of a local client device displays an avatar animation video in area 510 and a video of a caller from a remote device in area 520. The UI 500 includes a user-selectable element 530 indicating that an avatar communication session is ongoing, a user-selectable element 540 indicating that video is disabled at the local client device, and a user-selectable element 550 operable to end the video call session.

[0065] Machine Learning Program

[0066] Figure 6 The training and use of machine learning programs according to some examples are shown. Machine learning (ML) explores the study and construction of algorithms (also referred to herein as tools) that can learn from existing data and make predictions on new data. Such machine learning algorithms operate by building an ML model 616 using training data 612 in order to make data-driven predictions or decisions represented as outputs or evaluations 620.

[0067] Training an ML algorithm includes analyzing large amounts of data (e.g., from several gigabytes to a terabyte or more) to find data correlations. The ML algorithm uses training data 612 to find correlations between identified features 602 that influence outcomes or assessments 620. The training data 612 includes examples of values ​​for the features 602. The machine learning algorithm uses training data 612 to find correlations between identified features 602 that influence outcomes. A feature 602 is a single measurable attribute of an observed phenomenon. During training 614, the ML algorithm analyzes the training data 612 based on the identified features 602 and configuration parameters 611 defined for training. The result of training 614 is an ML model 616 that can accept input to produce an assessment.

[0068] Many ML algorithms include configuration parameters 611, and the more complex the ML algorithm, the more parameters are available to the user. Configuration parameters 611 define the variables used for the ML algorithm when searching for the best ML model. Training parameters include model parameters and hyperparameters. Model parameters are learned from training data, and hyperparameters are not learned from training data, but are provided to the ML algorithm. Some examples of model parameters include maximum model size, maximum number of passes through training data, data shuffling type, regression coefficients, decision tree split positions, etc. Hyperparameters may include the number of hidden layers in a neural network, the number of hidden nodes in each layer, the learning rate (possibly with various adaptation schemes for the learning rate), regularization parameters, the type of nonlinear activation function, etc. Finding the right (or optimal) set of hyperparameters can be a very time-consuming task, which requires the use of a large amount of computer resources. When performing an evaluation using the ML model 616, new data 618 is provided as input to the ML model 616, and the ML model 616 generates an evaluation 620 as output.

[0069] In some examples, the ML model 616 is provided with the avatar call platform and is configured to generate avatar animation data based on local call data. In one example, the ML model 616 estimates corresponding values ​​for cues describing the shape and positioning of facial features on a facial object of the avatar image. In this example, the new data 618 includes local data captured while the video call session is ongoing and the avatar communication session is activated. The training data used to train the ML model 616 includes images representing faces depicting facial expressions with relevant cues.

[0070] Machine structure

[0071] Figure 7700 within which instructions 708 (e.g., software, programs, applications, applet, app, or other executable code) may be executed for causing the machine 700 to perform any one or more of the methods discussed herein. For example, the instructions 708 may cause the machine 700 to perform any one or more of the methods described herein. The instructions 708 convert a general-purpose, non-programmed machine 700 into a specific machine 700 that is programmed to perform the functions described and illustrated in the manner described. The machine 700 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 700 may operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 700 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing instructions 708 specifying actions to be taken by the machine 700. In addition, although only a single machine 700 is shown, the term "machine" should also be considered to include a collection of machines that individually or jointly execute instructions 708 to perform any one or more of the methods discussed herein. For example, the machine 700 may include any of the client device 102 or a plurality of server devices that form part of the message server system 108. In some examples, the machine 700 may also include both a client system and a server system, wherein certain operations of a particular method or algorithm are performed on the server side and certain operations of the particular method or algorithm are performed on the client side.

[0072] The machine 700 may include a processor 702, a memory 704, and an input / output I / O component 738, which may be configured to communicate with each other via a bus 740. In an example, the processor 702 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 706 that executes instructions 708 and a processor 710. The term "processor" is intended to include a multi-core processor, which may include two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously. Although Figure 7 Multiple processors 702 are shown, but machine 700 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0073] The memory 704 includes a main memory 712, a static memory 714, and a storage unit 716, which are all accessible by the processor 702 via the bus 740. The main memory 704, the static memory 714, and the storage unit 716 store instructions 708 that embody any one or more of the methods or functions described herein. The instructions 708 may also reside, completely or partially, within the main memory 712, within the static memory 714, within the machine-readable medium 718 within the storage unit 771, within at least one of the processors 702 (e.g., within a cache memory of a processor), or within any suitable combination thereof during execution thereof by the machine 700.

[0074] I / O components 738 may include various components for receiving input, providing output, generating output, sending information, exchanging information, capturing measurements, etc. The specific I / O components 738 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It should be understood that I / O components 738 may include Figure 7720 , and the like. In various examples, the I / O components 738 may include a user output component 724 and a user input component 726. The user output component 724 may include a visual component (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), an auditory component (e.g., a speaker), a tactile component (e.g., a vibration motor, a resistance mechanism), other signal generators, etc. The user input component 726 may include an alphanumeric input component (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input component), a pointing-based input component (e.g., a mouse, a touch pad, a trackball, a joystick, a motion sensor, or another pointing instrument), a tactile input component (e.g., a physical button, a touch screen or other tactile input component that provides location and force of a touch or touch gesture), an audio input component (e.g., a microphone), etc.

[0075] In another example, the I / O component 738 may include a biometric component 728, a motion component 730, an environmental component 732, or a position component 734, as well as a variety of other components. For example, the biometric component 728 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 730 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, and a rotation sensor component (e.g., a gyroscope).

[0076] Environmental component 732 includes, for example, one or more cameras (with still imaging / video and video capture capabilities), an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor that detects concentrations of hazardous gases for safety or measures pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment.

[0077] With respect to cameras, client device 102 can have a camera system including, for example, a front-facing camera on a front surface of client device 102 and a rear-facing camera on a rear surface of client device 102. The front-facing camera can, for example, be used to capture still images and videos (e.g., “selfies”) of a user of client device 102, which can then be enhanced using the enhancement data (e.g., filters) described above. The rear-facing camera can, for example, be used to capture still images and videos in a more traditional camera mode, where the images are similarly enhanced using the enhancement data. In addition to the front-facing camera and the rear-facing camera, client device 102 can also include a 360° camera for capturing 360° photos and videos.

[0078] Additionally, the camera system of the client device 102 may include dual rear cameras (e.g., a main camera and a depth sensing camera), or even three, four, or five rear camera configurations on the front and back sides of the client device 102. For example, these multi-camera systems may include a wide-angle camera, an ultra-wide-angle camera, a telephoto camera, a macro camera, and a depth sensor.

[0079] The location component 734 includes a positioning sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure, from which the altitude can be obtained), an orientation sensor component (e.g., a magnetometer), etc.

[0080] Various technologies may be used to achieve communication. The I / O components 738 also include a communication component 736 that is operable to couple the machine 700 to the network 720 or device 722 via a corresponding coupling or connection. For example, the communication component 736 may include a network interface component or another suitable device for interfacing with the network 720. In other examples, the communication component 736 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, Parts (e.g. Low power consumption), Device 722 may be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via USB).

[0081] In addition, the communication component 636 can detect the identifier or include a component operable to detect the identifier. For example, the communication component 636 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional bar codes such as universal product codes (UPC) bar codes, multi-dimensional bar codes such as quick response (QR) codes, Aztec codes, data matrix, data symbols (Dataglyph), MaxiCode, PDF417, Ultra Code, UCC RSS-2D bar codes and other optical codes) or an acoustic detection component (e.g., a microphone for identifying tagged audio signals). In addition, various information can be obtained via the communication component 736, such as obtaining positioning via Internet Protocol (IP) geolocation, ... Positioning is obtained through signal triangulation, positioning is obtained via detection of NFC beacon signals that can indicate a specific position, etc.

[0082] Various memories (e.g., main memory 712, static memory 714, and memory of processor 702) and storage unit 716 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 708) when executed by processor 702 cause various operations to implement the disclosed examples.

[0083] Instructions 708 may be sent or received over network 720 using a transmission medium via a network interface device (e.g., a network interface component included in communications component 736) and using any of several well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instructions 608 may be sent or received via a coupling (e.g., a peer-to-peer coupling) with device 722 using a transmission medium.

[0084] Glossary

[0085] "Carrier signal" refers to any intangible medium that can store, encode or carry instructions for execution by the machine and includes digital or analog communications signals or other intangible media to facilitate communication of these instructions. Instructions may be sent or received over a network via a network interface device using a transmission medium.

[0086] "Client Device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communications device that a user may use to access a network.

[0087] "Communications network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, A network, other type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rate for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standard, other standards defined by various standard setting organizations, other long distance protocols, or other data transmission technologies.

[0088] "Component" refers to a device, a physical entity, or logic with boundaries defined by function or subroutine calls, branch points, APIs, or other technologies that provide partitioning or modularization for specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processing. Components can be packaged functional hardware units designed for use with other components and can be part of a program that generally performs specific functions in related functions. Components can constitute software components (e.g., codes implemented on machine-readable media) or hardware components. "Hardware components" are tangible units that can perform certain operations and can be configured or arranged in a certain physical manner. In various example embodiments, one or more computer systems (e.g., independent computer systems, client computer systems, or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., applications or application parts) to operate to perform certain operations as described herein. Hardware components can also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component may include a dedicated circuit or logic that is permanently configured to perform certain operations. The hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware component may also include a programmable logic or circuit that is temporarily configured to perform certain operations by software. For example, the hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine) that is uniquely customized to perform the configured function, and is no longer a general-purpose processor. It should be understood that the decision to implement the hardware component mechanically, in a dedicated and permanently configured circuit or in a temporarily configured (e.g., configured by software) circuit can be driven by cost and time considerations. Therefore, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in some way or perform certain operations described herein. Considering an embodiment in which the hardware component is temporarily configured (e.g., programmed), it is not necessary to configure or instantiate each hardware component in the hardware component at any time. For example, where a hardware component includes a general purpose processor that is configured by software to be a special purpose processor, the general purpose processor may be configured as different special purpose processors (e.g., including different hardware components) at different times. Thus, software configures one or more specific processors to, for example, constitute a specific hardware component at one time and to constitute different hardware components at different times. Hardware components may provide information to other hardware components and receive information from other hardware components. Thus, the described hardware components may be considered to be communicatively coupled.In the case of multiple hardware components being present at the same time, communication can be achieved by signal transmission (e.g., by appropriate circuits and buses) between or among two or more hardware components in the hardware components. In an embodiment in which multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessed by multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a memory device to which it is communicatively coupled. Then, other hardware components can access the memory device at a subsequent time to retrieve the stored output and process it. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (e.g., the collection of information). The various operations of the example methods described herein can be performed at least in part by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform related operations. Whether it is temporarily configured or permanently configured, such a processor can constitute a processor-implemented component that operates to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refer to hardware components implemented using one or more processors. Similarly, the method described herein can be implemented at least in part by a processor, wherein one or more specific processors are examples of hardware. For example, at least some of the operations in the operation of the method can be performed by one or more processors 704 or a component implemented by the processor. In addition, one or more processors can also operate to support the execution of related operations in a "cloud computing" environment or operate as "software as a service" (SaaS). For example, at least some of the operations in the operation can be performed by a computer group (as an example of a machine including a processor), wherein these operations can be accessed via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., API). The execution of certain operations can be distributed between processors, not only resident in a single machine, but deployed across multiple machines. In some example embodiments, a processor or a component implemented by a processor can be located in a single geographic location (e.g., in a home environment, an office environment, or a server cluster). In other example embodiments, a processor or a component implemented by a processor can be distributed across multiple geographic locations.

[0089] "Computer-readable storage media" refers to both machine storage media and transmission media. Thus, the term includes both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" mean the same thing and may be used interchangeably in this disclosure.

[0090] “Machine storage media” refers to a single or multiple storage devices and media (e.g., centralized or distributed databases, and associated caches and servers) that store executable instructions, routines, and data. Thus, the term should be considered to include, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms “machine storage media,” “device storage media,” and “computer storage media” mean the same thing and may be used interchangeably in this disclosure. The terms “machine storage media,” “computer storage media,” and “device storage media” expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are encompassed by the term “signal media.”

[0091] “Non-transitory computer-readable storage medium” refers to tangible media capable of storing, encoding, or carrying instructions to be executed by a machine.

[0092] "Signal medium" refers to any intangible medium that is capable of storing, encoding or carrying instructions executed by a machine, and "signal medium" includes digital or analog communication signals or other intangible media to facilitate the transfer of software or data. The term "signal medium" should be deemed to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode the information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0093] The present invention may also be configured as follows:

[0094] 1. A method comprising:

[0095] In a messaging system for exchanging data over a network, in a video call session between a first client device and a second client device, activating an avatar communication process for the first client device, the avatar communication process comprising:

[0096] receiving, at the first client device, local call data and generating avatar animation data based on the local call data, the local call data originating from the first client device, the avatar animation data encoding movement of one or more facial features;

[0097] generating an avatar animation video based on the avatar animation data and the avatar image; and

[0098] The avatar animation video is displayed on a display device.

[0099] 2. The method according to item 1, comprising:

[0100] transmitting the avatar animation data along with audio data from the local call data to the second client device;

[0101] generating, at the second client device, the avatar animation video based on the avatar animation data and the avatar image; and

[0102] The avatar animation video is caused to be displayed at the second client device.

[0103] 3. The method according to item 1, comprising:

[0104] transmitting a compressed version of the avatar animation video along with audio data from the local call data to the second client device;

[0105] generating, at the second client device, the avatar animation video based on the avatar animation data and the avatar image; and

[0106] The avatar animation video is caused to be displayed at the second client device.

[0107] 4. The method according to 1, wherein the local call data includes a video, and generating the avatar animation data includes:

[0108] detecting facial objects in the video; and

[0109] The avatar animation data is generated based on features of the facial object in the video.

[0110] 5. The method of claim 1, wherein generating the avatar animation data is based on features of audio data from the local call data.

[0111] 6. The method of claim 1, wherein the video call session is associated with a user profile representing a user in the messaging system, and wherein the avatar image is a cartoon representation of the user in the messaging system.

[0112] 7. The method of claim 1, comprising detecting, at the first client device, a request to activate the avatar communication process, wherein activating the avatar communication process is responsive to detecting the request.

[0113] 8. The method according to item 1, comprising:

[0114] detecting, at the first client device, a request to disable video in the video call session;

[0115] presenting a user-selectable element operable to activate said avatar communication process;

[0116] detecting activation of the user-selectable element; and

[0117] The avatar communication process is activated.

[0118] 9. The method of 8, comprising generating a user interface comprising the user-selectable element operable to activate the avatar communication process.

[0119] 10. The method of claim 1, wherein generating the avatar animation data comprises executing a neural network machine learning model.

[0120] 11. A system comprising:

[0121] one or more processors; and

[0122] A non-transitory computer-readable storage medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

[0123] In a messaging system for exchanging data over a network, in a video call session between a first client device and a second client device, activating an avatar communication process for the first client device, the avatar communication process comprising:

[0124] receiving, at the first client device, local call data and generating avatar animation data based on the local call data, the local call data originating from the first client device, the avatar animation data encoding movement of one or more facial features;

[0125] generating an avatar animation video based on the avatar animation data and the avatar image; and

[0126] The avatar animation video is displayed on a display device.

[0127] 12. The system of claim 11, wherein the operations caused by the instructions executed by the one or more processors further comprise:

[0128] transmitting the avatar animation data along with audio data from the local call data to the second client device;

[0129] generating, at the second client device, the avatar animation video based on the avatar animation data and the avatar image; and

[0130] The avatar animation video is caused to be displayed at the second client device.

[0131] 13. The system of claim 11, wherein the operations caused by the instructions executed by the one or more processors further comprise:

[0132] transmitting a compressed version of the avatar animation video along with audio data from the local call data to the second client device;

[0133] generating, at the second client device, the avatar animation video based on the avatar animation data and the avatar image; and

[0134] The avatar animation video is caused to be displayed at the second client device.

[0135] 14. The system of claim 11, wherein the local call data comprises a video, generating the avatar animation data, wherein the operations caused by the instructions executed by the one or more processors further comprise:

[0136] detecting facial objects in the video; and

[0137] The avatar animation data is generated based on features of the facial object in the video.

[0138] 15. The system of claim 11, wherein generating the avatar animation data is based on features of audio data from the local call data.

[0139] 16. The system of claim 11, wherein the video call session is associated with a user profile representing a user in the messaging system, the avatar image being a cartoon representation of the user in the messaging system.

[0140] 17. The system of claim 11, wherein the operations caused by the instructions executed by the one or more processors further comprise: detecting, at the first client device, a request to activate the avatar communication process, wherein activating the avatar communication process is in response to detecting the request.

[0141] 18. The system of claim 11, wherein the operations caused by the instructions executed by the one or more processors further comprise:

[0142] detecting, at the first client device, a request to disable video in the video call session;

[0143] presenting a user-selectable element operable to activate said avatar communication process;

[0144] detecting activation of the user-selectable element; and

[0145] The avatar communication process is activated.

[0146] 19. The system of claim 18, wherein the operations caused by the instructions executed by the one or more processors further comprise generating a user interface comprising the user-selectable element operable to activate the avatar communication process.

[0147] 20. A machine-readable non-transitory storage medium having instruction data executable by a machine to cause the machine to perform operations, the operations comprising:

[0148] In a messaging system for exchanging data over a network, in a video call session between a first client device and a second client device, activating an avatar communication process for the first client device, the avatar communication process comprising:

[0149] receiving, at the first client device, local call data and generating avatar animation data based on the local call data, the local call data originating from the first client device, the avatar animation data encoding movement of one or more facial features;

[0150] generating an avatar animation video based on the avatar animation data and the avatar image; and

[0151] The avatar animation video is displayed on a display device.

Claims

1. A method for enhancing a video call experience, comprising: presenting, at a first device, a first user-selectable element operable to inhibit video in a video call session between the first device and a second device; in response to detecting activation of the first user-selectable element, presenting at the first device a second user-selectable element operable to activate an avatar communication process; as well as In response to detecting activation of the second user-selectable element, activating an avatar communication process for the first device, the avatar communication process comprising: generating an avatar animation video based on avatar animation data, the avatar animation data comprising a plurality of data points representing positions of facial features in a region representing a face; as well as The avatar animation video is displayed on a display.

2. The method according to claim 1, wherein: The avatar communication process further includes: analyzing the local call data to generate information representative of the motion of the facial feature, the information comprising the position of the facial feature relative to an expressionless position of the facial feature; and A plurality of video frames are generated based on the motion of the facial feature, each video frame being generated based on a position of the facial feature relative to the expressionless position of a corresponding portion of the motion.

3. The method according to claim 1, wherein: The avatar communication process further includes: Objects including facial objects detected in local call data including a video feed are tracked, and the avatar animation video is generated based on features of the objects.

4. The method according to claim 1, wherein: The plurality of data points representing the position of the facial feature describes the position of the facial feature as a number of points away from an expressionless position of the facial feature.

5. The method according to claim 1, wherein: The avatar communication process further includes: generating a compressed version of the avatar animation video; and A compressed version of the avatar animation video is transmitted to the second device.

6. The method according to claim 1, further comprising: The real-time video of the user of the second device and the avatar animation video are displayed on the display simultaneously.

7. The method according to claim 1, further comprising: In response to detecting activation of the second user-selectable element, presenting at the first device a third user-selectable element indicating that the avatar communication process is active, and the third user-selectable element is operable to end the avatar communication process.

8. The method according to claim 1, wherein: When the camera of the first device is turned on, the second user-selectable element is presented which is operable to activate the avatar communication process.

9. The method according to claim 1, further comprising: A third user-selectable element operable to end the video call session is presented at the first device.

10. The method according to claim 1, wherein: The avatar animation video is generated based on an avatar image selected from a plurality of avatar images, each avatar image being associated with a corresponding state or activity.

11. A system for enhancing video call experience, comprising: one or more processors; as well as A non-transitory computer-readable storage medium comprising instructions that, when executed by the one or more processors, cause the system to perform operations comprising: presenting, at a first device, a first user-selectable element operable to inhibit video in a video call session between the first device and a second device; In response to detecting activation of the first user-selectable element, presenting at the first device a second user-selectable element operable to activate an avatar communication process; and In response to detecting activation of the second user-selectable element, activating an avatar communication process for the first device, the avatar communication process comprising: generating an avatar animation video based on avatar animation data, the avatar animation data comprising a plurality of data points representing positions of facial features in an area representing a face; and The avatar animation video is displayed on a display.

12. The system according to claim 11, wherein: The avatar communication process further includes: analyzing the local call data to generate information representative of the motion of the facial feature, the information comprising the position of the facial feature relative to an expressionless position of the facial feature; and A plurality of video frames are generated based on the motion of the facial feature, each video frame being generated based on a position of the facial feature relative to the expressionless position of a corresponding portion of the motion.

13. The system according to claim 11, wherein: The avatar communication process further includes: Objects including facial objects detected in local call data including a video feed are tracked, and the avatar animation video is generated based on features of the objects.

14. The system according to claim 11, wherein: The plurality of data points representing the position of the facial feature describes the position of the facial feature as a number of points away from an expressionless position of the facial feature.

15. The system according to claim 11, wherein: The avatar communication process further includes: generating a compressed version of the avatar animation video; and A compressed version of the avatar animation video is transmitted to the second device.

16. The system of claim 11, the operations further comprising: The real-time video of the user of the second device and the avatar animation video are displayed on the display simultaneously.

17. The system of claim 11, the operations further comprising: In response to detecting activation of the second user-selectable element, presenting at the first device a third user-selectable element indicating that the avatar communication process is active, and the third user-selectable element is operable to end the avatar communication process.

18. The system of claim 11, wherein: When the camera of the first device is turned on, the second user-selectable element is presented which is operable to activate the avatar communication process.

19. The system of claim 11, the operations further comprising: A third user-selectable element operable to end the video call session is presented at the first device.

20. A non-transitory machine-readable storage medium having instruction data executable by a machine to cause the machine to perform operations comprising: presenting, at a first device, a first user-selectable element operable to inhibit video in a video call session between the first device and a second device; in response to detecting activation of the first user-selectable element, presenting at the first device a second user-selectable element operable to activate an avatar communication process; as well as In response to detecting activation of the second user-selectable element, activating an avatar communication process for the first device, the avatar communication process comprising: generating an avatar animation video based on avatar animation data, the avatar animation data comprising a plurality of data points representing positions of facial features in a region representing a face; as well as The avatar animation video is displayed on a display.

Images