Location sharing using different rate-constrained links

Abstract

Aspects of the present disclosure relate to systems and methods for performing operations, the operations comprising: storing multiple real-time graph (RTG) instances comprising multiple device objects on a distributed storage system; receiving a status update by a first device object among the multiple device objects from a client device associated with a first user; sending a first message comprising the status update by the first device object to a second device object associated with a second user via a real-time link; and sending a second message comprising the status update by the first device object to a third device object associated with a third user via a first rate-limited link.

Description

[0001] Priority claim

[0002] This application claims the benefit of priority to U.S. patent application Ser. No. 16 / 946,286, filed Jun. 15, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure generally relates to real-time location sharing. Background Art

[0004] The popularity of users interacting with other users online continues to grow. There are many ways for users to interact with other users online. Users can communicate with their friends using messaging applications, and can play multiplayer video games online with other users, or perform other actions using various other applications. Users also increasingly want to see where their friends are currently located. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] In the drawings, which are not necessarily drawn to scale, the same reference numerals may describe similar components in different views. To easily identify the discussion of any particular element or action, one or more of the highest digits in the reference numerals refer to the figure number in which the element is first introduced. Some examples are shown by way of example and not limitation in the accompanying drawings, in which:

[0006] Figure 1 is a diagrammatic representation of a networked environment in which the present disclosure may be deployed, according to some examples.

[0007] Figure 2 is a diagrammatic representation of a messaging system having both client-side and server-side functionality according to some examples.

[0008] Figure 3 is a diagrammatic representation of data structures maintained in a database according to some examples.

[0009] Figure 4 is a graphical representation of messages according to some examples.

[0010] Figure 5 is a diagrammatic representation of a distributed real-time status update server according to some examples.

[0011] Figure 6 is another diagrammatic representation of a distributed real-time status update server according to some examples.

[0012] Figure 7 is a diagrammatic representation of an RTG instance according to some examples.

[0013] FIG. 8A to FIG. 8Cis a diagrammatic representation of message exchanges between device objects of a distributed real-time status update server according to some examples.

[0014] Fig. 9 is a graphical representation of sending updates based on different friends list versions according to some examples.

[0015] FIG. 10A to FIG. 10D is a flow chart illustrating example operation of a distributed real-time status update server according to an example.

[0016] Fig.11 is a diagrammatic representation of a machine in the form of a computer system according to some examples, within which a set of instructions may be executed, for causing the machine to perform any one or more of the methodologies discussed herein.

[0017] Fig.12 is a block diagram illustrating a software architecture in which examples may be implemented. DETAILED DESCRIPTION

[0018] The following description includes systems, methods, techniques, instruction sequences and computing machine program products that implement the illustrative examples of the present disclosure. In the following description, for the purpose of illustration, many specific details are set forth to provide an understanding of various examples. However, it will be apparent to those skilled in the art that examples may be practiced without these specific details. Typically, known instruction instances, protocols, structures and techniques need not be shown in detail.

[0019] Typically, users not only access messaging applications to communicate with their friends, but also access messaging applications to determine the current location of their friends. Some messaging applications present the current location as part of a chat interface in which users exchange messages, and may also present the location on an interactive map. For example, an avatar representing a user may be located at the corresponding location of each user on a map. Obtaining the current location of a friend consumes a lot of computing resources and network resources. Specifically, the location of each friend may be pushed to a specific user device that needs to exchange a large number of messages over the network. This may degrade network performance and unnecessarily consume network bandwidth.

[0020] In some systems, a friend's location may be polled by a specific user device. For example, a user device may send a request with an identifier of one or more friends to a server to receive the current location of one or more friends. However, even in these systems, network bandwidth and system resources are wasted. Specifically, sometimes the location of a friend requested by a given user device may not change very frequently, and polling the location of such a friend causes the same location to be sent back to a given user device, which wastes system resources. In addition, a delay is introduced due to polling or pushing the location of a friend to various users, which reduces the accuracy of the location being transmitted. That is, for example, if a given friend is actively moving around, the location transmitted to the requesting device may not reflect the actual current location of the friend. This makes such a system less attractive and reduces the overall user experience.

[0021] The disclosed example seeks to improve the efficiency and speed of providing status updates such as location information to client devices in real time using a distributed storage system. The distributed storage system uses a real-time map instance to store device objects representing status information (e.g., location information) of each user. The device objects communicate with each other through a front-end instance. When the location of a given user changes, the location is updated in the user's device object. The location is propagated to the device objects of the user's friends in an intelligent and automatic manner. When a given friend of the user's friends requests the user's current location, such as when a given friend opens an interactive map of an avatar, the given friend's device communicates with the front-end instance to retrieve the location of the user's device stored in the given friend's device object. This reduces the total network traffic and increases the rate at which location updates are delivered and exchanged. In this way, the lag between when the location is requested and when the location is provided to a given user is minimized, which provides a real-time location sharing experience for the user.

[0022] Specifically, according to the disclosed example, a front-end (FE) instance and a plurality of real-time graph (RTG) instances are stored on a distributed storage system. In some cases, a plurality of FE instances and a plurality of RTG instances may be stored, the FE instances may be proportionally increased / decreased based on the load and number of client devices, and the RTG instances have a fixed size. Each of the plurality of RTG instances includes a plurality of device objects, and the FE instance is configured to communicate with a client device associated with a first user. A bidirectional streaming remote procedure call (RPC) connection is established between the FE instance and the plurality of RTG instances. The FE instance receives a status update from the client device, and determines that a first device object corresponding to the client device is stored on a first RTG instance of the plurality of RTG instances. The FE instance sends a first message including a status update to the first RTG instance to update the first device object. The first device object identifies a friend of the user, and provides a status update to the device object of the friend of the user. When a given friend of the user eventually requests a status update such as a user's location, the information is readily available in the device object of the given friend, and is delivered to the device of the given friend in real time.

[0023] In this manner, according to the disclosed examples, device resources (eg, processor cycles, memory, and power usage) required to complete tasks with the device are reduced.

[0024] Networked computing environment

[0025] Figure 1 1 is a block diagram illustrating an example messaging system 100 for exchanging data (e.g., messages and associated content) over a network. The messaging system 100 includes multiple instances of client devices 102, each hosting multiple applications including a messaging client 104. Each messaging client 104 is communicatively coupled to a messaging server system 108 and other instances of the messaging client 104 via a network 106 (e.g., the Internet). The messaging server system 108 may include a distributed real-time status update server 107, in which case each messaging client 104 receives status updates (e.g., location updates, location preference updates, online status updates, etc.) from the distributed real-time status update server 107 via the messaging server system 108. In some implementations, all or a portion of the distributed real-time status update server 107 is implemented external to the messaging server system 108. In these cases, each messaging client 104 receives status updates (e.g., location updates, location preference updates, online status updates, etc.) directly from the distributed real-time status update server 107 via the network 106.

[0026] The messaging client 104 is able to communicate and exchange data with another messaging client 104 and a messaging server system 108 via the network 106. The data exchanged between the messaging clients 104 and between the messaging clients 104 and the messaging server system 108 includes functions (e.g., commands to activate functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0027] The messaging server system 108 provides server-side functionality to specific messaging clients 104 via the network 106. Although certain functions of the messaging system 100 are described herein as being performed by the messaging client 104 or by the messaging server system 108, the location of certain functions within the messaging client 104 or messaging server system 108 may be a design choice. For example, it may be technically preferred to initially deploy certain technologies and functions within the messaging server system 108, but later migrate the technologies and functions to the messaging client 104 where the client device 102 has sufficient processing power.

[0028] The messaging server system 108 supports various services and operations provided to the messaging clients 104. Such operations include sending data to the messaging clients 104, receiving data from the messaging clients 104, and processing data generated by the messaging clients 104. As examples, the data may include message content, client device information, geo-location information, media enhancements and overlays, message content persistence conditions, social networking information, and live event information. Data exchanges within the messaging system 100 are activated and controlled by functions available via the user interface (UI) of the messaging clients 104.

[0029] Turning now specifically to the messaging server system 108, an application program interface (API) server 110 is coupled to and provides a programming interface to an application server 112. The application server 112 is communicatively coupled to a database server 118, which facilitates access to a database 120 that stores data associated with messages processed by the application server 112. Similarly, a web server 124 is coupled to and provides a web-based interface to the application server 112. To this end, the web server 124 processes incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0030] The application program interface (API) server 110 receives and sends message data (e.g., commands and message payloads) between the client device 102 and the application server 112. Specifically, the application program interface (API) server 110 provides a set of interfaces (e.g., routines and protocols) that the messaging client 104 can call or query to activate the functions of the application server 112. The application program interface (API) server 110 exposes various functions supported by the application server 112, including: account registration; login functionality; sending messages from a particular messaging client 104 to another messaging client 104 via the application server 112; sending media files (e.g., images or videos) from a messaging client 104 to the messaging server 114 and for possible access by another messaging client 104; setting up a collection of media data (e.g., a story); retrieving a friend list of a user of the client device 102; retrieving such a collection; retrieving messages and content; adding and removing entities (e.g., friends) from an entity graph (e.g., a social graph); locating friends in a social graph; and opening application events (e.g., related to a messaging client 104).

[0031] The application server 112 hosts several server applications and subsystems, including, for example, a messaging server 114, an image processing server 116, and a social networking server 122. The messaging server 114 implements several message processing techniques and functions, particularly those related to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client 104. As will be described in further detail, text and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or galleries). These collections are then made available to the messaging client 104. In view of the hardware requirements for such processing, other processor- and memory-intensive processing of data can also be performed on the server side by the messaging server 114.

[0032] The application server 112 also includes an image processing server 116 that is dedicated to performing various image processing operations, typically with respect to images or videos within the payload of messages sent from or received at the messaging server 114 .

[0033] The social network server 122 supports various social networking functions and services and makes these functions and services available to the messaging server 114. To this end, the social network server 122 maintains and accesses the entity graph 306 (e.g., Figure 3). Examples of functions and services supported by social network server 122 include identifying other users of messaging system 100 with whom a particular user has relationships or "follows" him, and identifying interests and other entities of a particular user.

[0034] The distributed real-time status update server 107 stores a real-time graph (RTG) of instances or nodes. These nodes maintain various information about different users of the messaging server 114. The client device 102 receives updates (e.g., location updates) from friends in real time from the distributed real-time status update server 107. The distributed real-time status update server 107 includes one or more front-end (FE) instances and one or more real-time graph (RTG) instances. When a given client device 102 is initially connected to the distributed real-time status update server 107, or when a message is directed to a user of a given client device 102, a FE instance is created for the client device 102 on the distributed real-time status update server 107 and the FE instance is stored. In some cases, an already existing FE instance is accessed, in which case, an FE instance does not need to be created when the client device 102 is connected. This occurs when the client device 102 has previously established a streaming connection to an FE instance or another FE instance. FE instances are created on demand according to the load of the server 107. This can include storing a unique device identifier of a user of a client device or a client device in a FE instance. The FE instance establishes a real-time link to a given client device 102 for exchanging information with the client device 102. The FE instance also establishes a bidirectional streaming real-time gRPC (remote procedure call) connection with each of the multiple RTG instances.

[0035] In some examples, the FE instance receives an update (e.g., a location update) from the client device 102. In response, the FE instance identifies a device object corresponding to the client device 102. Specifically, each RTG instance stores one or more device objects corresponding to each client device 102 (e.g., a physical device). The FE instance accesses routing information that identifies which of the multiple RTG instances stores the device object for the client device 102. The FE instance then sends a status update to the RTG instance that stores the device object for the client device 102 via the previously established gRPC connection.

[0036] The RTG instance receives a status update (e.g., a location update or an update to preference information such as a location sharing preference) and identifies a device object to be updated for the client device 102. The RTG instance updates the corresponding status update stored in the device object with the received update. The device object processes the update and identifies multiple friends associated with the user of the client device 102 on the messaging system 100. The device object generates multiple messages directed to the device objects of friends associated with the user, for example, based on the current state of the corresponding device object (e.g., the device object indicates whether the friend is active or inactive). The message includes an update to the status (e.g., an update to the location). In some cases, the device object first determines whether the status update meets certain criteria. For example, the device object determines whether the location included in the update differs from the previously stored location by more than a threshold amount. As another example, the device object determines whether the location included in the update was last updated before exceeding a threshold amount of time (e.g., the timestamp of the previously stored location exceeds the current timestamp of the location in the update by more than a specified amount). As another example, the device object determines whether the location included in the update is to be shared with the user based on the user's current status (e.g., whether the user is currently active or inactive, whether the user is currently reading a message or accessing an interactive map presenting the locations of friends in real time, the user's last connection time, etc.). If the status update meets the criteria or criteria, the device object sends the message to the friends' device objects. In some cases, the criteria may be met for a first group of friends, but not for a second group of friends. In this case, the device object sends the status update to the device objects of the first group of friends and delays or completely blocks the sending of updates to the device objects of the second group of friends.

[0037] In some examples, a given friend's device object resides or is stored on an RTG instance (e.g., a second RTG instance) that is different from the RTG instance that stores the user's device object. In this case, the user's device object sends a message with an update directed to the friend's device object to a randomly or pseudo-randomly selected FE instance. In some cases, the FE instances are selected in a round-robin manner, where the FE instances are stored in a list, and the next FE instance adjacent to the FE instance to which the message was previously sent is selected. Once the last FE instance in the list is reached, the first FE instance in the list is selected to send a message to the device object. The FE instance receives the message and determines which of the multiple RTG instances stores the given friend's device object. The FE instance uses the routing information stored in the FE instance to identify the second RTG instance that stores the device object. The FE instance then sends the message to the RTG instance that stores the device object. The RTG instance receives the message and updates the user's status stored in the friend's device object.

[0038] After a threshold time interval after a given client device 102 is disconnected from the server 107, the device object stored on the RTG instance is automatically deleted from the distributed real-time status update server 107. Once the threshold time interval is reached, the information stored in the device object is moved to a different persistent storage device, and the real-time connection with the client device 102 and the gRPC connection to the RTG instance are terminated. In this way, whenever a given friend has an update to their status, the update is propagated to the device object of the user associated with the friend. When such a user is finally connected to the system (e.g., opening the messaging client 104), the status information stored in the device object of such a user is made available to the user seamlessly and quickly. This avoids the need for users to poll or wait to receive status updates directly from their friends, because such information is updated in the user's device object when the user is offline. In some cases, FE instances are created and deleted based on the load of the server 107. A gRPC proxy object is generated and stored on the FE instance to communicate with the client device 102 via a gRPC connection. Such a proxy object provides information such as status updates or information to a corresponding device object stored on the RTG instance from the client device 102. Once the connection between the gRPC proxy object and the client device 102 is terminated, the gRPC proxy object is deleted and the FE instance remains stored and available to handle further connections.

[0039] System Architecture

[0040] Figure 2 1 is a block diagram showing further details about the messaging system 100 according to some examples. Specifically, the messaging system 100 is shown to include a messaging client 104 and an application server 112. The messaging system 100 contains several subsystems that are supported on the client side by the messaging client 104 and on the server side by the application server 112. These subsystems include, for example, a transient timer system 202, a collection management system 204, an enhancement system 206, a mapping system 208, and a gaming system 210.

[0041] The transient timer system 202 is responsible for enforcing temporary or time-limited access to content by the messaging client 104 and the messaging server 114. The transient timer system 202 includes a number of timers that selectively enable access (e.g., for presentation and display) of messages and associated content via the messaging client 104 based on duration and display parameters associated with a message or collection of messages (e.g., a story). Additional details regarding the operation of the transient timer system 202 are provided below.

[0042] The collection management system 204 is responsible for managing groups or collections of media (e.g., collections of text, images, video, and audio data). Collections of content (e.g., messages, including images, videos, text, and audio) can be organized into "event libraries" or "event stories." Such collections can be available for a specified period of time (e.g., the duration of an event related to the content). For example, content related to a concert can be made available as a "story" for the duration of the concert. The collection management system 204 can also be responsible for publishing an icon to the user interface of the messaging client 104 that provides notification that a particular collection exists.

[0043] In addition, the collection management system 204 also includes a curation interface 212 that enables collection managers to manage and curate specific content collections. For example, the curation interface 212 enables event organizers to curate content collections related to specific events (e.g., remove inappropriate content or redundant messages). In addition, the collection management system 204 uses machine vision (or image recognition technology) and content rules to automatically curate content collections. In some examples, users can be paid compensation for including user-generated content in a collection. In such cases, the collection management system 204 operates to automatically pay such users for using their content.

[0044] The enhancement system 206 provides various functions that enable a user to enhance (e.g., mark or otherwise modify or edit) media content associated with a message. For example, the enhancement system 206 provides functions related to generating and publishing media overlays for messages processed by the messaging system 100. The enhancement system 206 is operable to provide a media overlay or enhancement (e.g., an image filter) to the messaging client 104 based on the geographic location of the client device 102. In another example, the enhancement system 206 is operable to supply a media overlay to the messaging client 104 based on other information such as social network information of the user of the client device 102. The media overlay may include audio and visual content and visual effects. Examples of audio and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects may be applied to media content items (e.g., photos) at the client device 102. For example, a media overlay may include text or images that may be overlaid on a photo taken by the client device 102. In another example, the media overlay includes a location identification overlay (e.g., Venice Beach), the name of a real-time event, or a business name overlay (e.g., Beach Cafe). In another example, the enhancement system 206 uses the geographic location of the client device 102 to identify a media overlay that includes the name of the business at the geographic location of the client device 102. The media overlay may include other tags associated with the business. The media overlay may be stored in the database 120 and accessed by the database server 118.

[0045] In some examples, the enhancement system 206 provides a user-based publishing platform that enables a user to select a geographic location on a map and upload content associated with the selected geographic location. The user can also specify situations in which a particular media overlay should be provided to other users. The enhancement system 206 generates a media overlay that includes the uploaded content and associates the uploaded content with the selected geographic location.

[0046] In other examples, the enhancement system 206 provides a merchant-based publishing platform that enables merchants to select specific media coverage associated with a geographic location via a bidding process. For example, the enhancement system 206 associates the media coverage of the highest bidding merchant with the corresponding geographic location for a predefined amount of time.

[0047] The mapping system 208 provides various geolocation functions and supports the presentation of map-based media content and messages by the messaging client 104. For example, the mapping system 208 enables the display of a user icon or avatar (e.g., stored in the profile data 308) on a map to indicate the current or past locations of the user's "friends" within the context of the map, as well as media content (e.g., a collection of messages including photos and videos) generated by these friends. For example, a message posted by a user to the messaging system 100 from a particular geographic location can be displayed to the "friends" of a particular user at the particular location within the context of the map on the mapping interface of the messaging client 104. The user can also share his or her location and status information with other users of the messaging system 100 (e.g., using an appropriate status avatar) via the messaging client 104, where the location and status information is similarly displayed to the selected user within the context of the mapping interface of the messaging client 104.

[0048] The gaming system 210 provides various gaming functions within the context of the messaging client 104. The messaging client 104 provides a gaming interface that provides a list of available games (e.g., web-based games or web-based applications) that can be launched by a user in the context of the messaging client 104 and played with other users of the messaging system 100. The messaging system 100 also enables a particular user to invite other users to participate in playing a particular game by sending an invitation from the messaging client 104 to such other users. The messaging client 104 also supports both voice messaging and text messaging (e.g., chatting) in the context of game play, provides leaderboards for games, and also supports providing in-game rewards (e.g., coins and items).

[0049] Data Architecture

[0050] Figure 3 is a diagram illustrating a data structure 300 that may be stored in a database 120 of a messaging server system 108 according to some examples. Although the contents of the database 120 are illustrated as including several tables, it should be understood that data may be stored in other types of data structures (e.g., as an object-oriented database).

[0051] The database 120 includes message data stored in a message table 302. For any particular message, the message data includes at least message sender data, message recipient (or receiver) data, and a payload. Figure 4 Additional details regarding information that may be included in a message and included in the message data stored in message table 302 are described.

[0052] The entity table 304 stores entity data and is linked (e.g., referenced) to the entity graph 306 and profile data 308. Entities for which records are maintained within the entity table 304 may include individuals, corporate entities, organizations, objects, places, events, etc. Regardless of the entity type, any entity about which the messaging server system 108 stores data may be an identified entity. Each entity is provided with a unique identifier as well as an entity type identifier (not shown).

[0053] The entity graph 306 stores information about relationships and associations between entities. As examples only, such relationships may be interest-based or activity-based social relationships, professional relationships (e.g., working in a common company or organization).

[0054] The profile data 308 stores multiple types of profile data about a particular entity. Based on the privacy settings specified by the particular entity, the profile data 308 can be selectively used and presented to other users of the messaging system 100. In the case where the entity is a person, the profile data 308 includes, for example, a user name, phone number, address, settings (e.g., notification and privacy settings), and an avatar representation (or a collection of such avatar representations) selected by the user. The particular user can then selectively include one or more of these avatar representations in the content of messages transmitted via the messaging system 100 and on a map interface displayed to other users by the messaging client 104. The collection of avatar representations can include a "status avatar" that presents a graphical representation of the state or activity that the user may choose to communicate at a particular time.

[0055] Where the entity is a group, the profile data 308 for the group may similarly include one or more avatar representations associated with the group, in addition to the group name, members, and various settings related to the group (eg, notifications).

[0056] Database 120 also stores enhancement data, such as overlays or filters, in enhancement table 310. Enhancement data is associated with and applied to videos (data for videos is stored in video table 314) and images (data for images is stored in image table 316).

[0057] In one example, a filter is an overlay that is displayed as an overlay on an image or video during presentation to a recipient user. The filters may be of various types, including filters selected by a user from a set of filters presented to a sending user by messaging client 104 while the sending user is composing a message. Other types of filters include geolocation filters (also referred to as geofilters), which may be presented to a sending user based on a geographic location. For example, a geolocation filter specific to a nearby or special location may be presented within a user interface by messaging client 104 based on geographic location information determined by a global positioning system (GPS) unit of client device 102.

[0058] Another type of filter is a data filter, which can be selectively presented to the sending user by the messaging client 104 based on other input or information collected during the message creation process by the client device 102. Examples of data filters include the current temperature at a particular location, the current speed the sending user is traveling, the battery life of the client device 102, or the current time.

[0059] Other augmented data that may be stored in the image table 316 include augmented reality content items (eg, corresponding to application lenses or augmented reality experiences). Augmented reality content items may be real-time special effects and sounds that may be added to an image or video.

[0060] As described above, augmented data includes augmented reality content items, overlays, image transformations, AR images, and similar terms referring to modifications that can be applied to image data (e.g., videos or images). This includes real-time modifications, which modify the image when it is captured using the device sensor (e.g., one or more cameras) of the client device 102 and then display the image on the screen of the client device 102 with the modification. This also includes modifications to stored content, such as modifications to video clips in a library that can be modified. For example, in a client device 102 that can access multiple augmented reality content items, a user can use a single video clip with multiple augmented reality content items to see how different augmented reality content items will modify the stored clips. For example, by selecting different augmented reality content items for the content, multiple augmented reality content items that apply different pseudo-random motion models can be applied to the same content. Similarly, real-time video capture can be used with the modifications shown to show how the video image currently captured by the sensor of the client device 102 will modify the captured data. Such data may simply be displayed on the screen without being stored in memory, or content captured by the device sensors may be recorded and stored in memory with or without modification (or both). In some systems, a preview function may show how different augmented reality content items will appear in different windows of the display simultaneously. For example, this may enable viewing multiple windows with different pseudo-random animations on the display simultaneously.

[0061] Thus, various systems using augmented reality content items or other such transformation systems and data that use the data to modify content may involve detection of objects (e.g., faces, hands, bodies, cats, dogs, surfaces, objects, etc.) in video frames, tracking of these objects as they leave the field of view, enter the field of view, and move around the field of view, and modification or transformation of these objects as they are tracked. In various examples, different methods for implementing such transformations may be used. Some examples may involve generating a 3-dimensional mesh model of one or more objects, and using transformations and animated textures of the model within the video to implement the transformation. In other examples, tracking of points on an object may be used to place an image or texture (which may be two-dimensional or three-dimensional) at the tracked location. In further examples, neural network analysis of video frames may be used to place images, models, or textures in content (e.g., images or video frames). Thus, augmented reality content items refer to both images, models, and textures used to create transformations in content, as well as additional modeling and analysis information required to implement such transformations through object detection, tracking, and placement.

[0062] Real-time video processing can be performed using any type of video data (e.g., video streams, video files, etc.) stored in the memory of any type of computerized system. For example, a user can load a video file and save it in the memory of the device, or a video stream can be generated using the device's sensors. In addition, any object, such as a human face and parts of a human body, an animal, or a non-living thing (such as a chair, a car, or other items) can be processed using computer animation models.

[0063] In some examples, when a specific modification is selected together with the content to be transformed, the element to be transformed is identified by the computing device, and then if the element to be transformed is present in the frame of the video, it is detected and tracked. The elements of the object are modified according to the modification request, thereby transforming the frame of the video stream. For different types of transformations, the transformation of the frame of the video stream can be performed by different methods. For example, for the transformation of the frame that mainly refers to the form of the elements of the object to be changed, the feature points of each element of the object (for example, using the active shape model (ASM) or other known methods) are calculated. Then, a grid based on the feature points is generated for each of at least one element of the object. The grid is used for the subsequent stage of tracking the elements of the object in the video stream. During the tracking process, the grid mentioned for each element is aligned with the position of each element. Then, additional points are generated on the grid. A set of first points is generated for each element based on the modification request, and a set of second points is generated for each element based on the set of first points and the request for modification. Then, the frame of the video stream can be transformed by modifying the elements of the object based on the set of first points and the set of second points and the grid. In this method, the background of the modified object can also be changed or distorted by tracking and modifying the background.

[0064] In some examples, a transformation that changes some areas of an object using elements of an object can be performed by calculating feature points of each element of the object and generating a grid based on the calculated feature points. Points are generated on the grid, and then various areas are generated based on these points. Then, the elements of the object are tracked by aligning the area of ​​each element with the position of each of at least one element, and the properties of the area can be modified based on the request for modification, thereby converting the frame of the video stream. Depending on the specific modification request, the properties of the mentioned area can be transformed in different ways. Such modifications may involve: changing the color of the area; removing at least some parts of the area from the frame of the video stream; including one or more new objects in the area based on the modification request; and modifying or deforming the elements of the area or object. In various examples, any combination of such modifications or other similar modifications may be used. For certain models to be animated, some feature points may be selected as control points for determining the entire state space of options for model animation.

[0065] In some examples of computer animation models that use face detection to transform image data, faces are detected on an image using a specific face detection algorithm (e.g., Viola-Jones). An active shape model (ASM) algorithm is then applied to the face region of the image to detect facial feature reference points.

[0066] In other examples, other methods and algorithms suitable for face detection can be used. For example, in some examples, landmarks are used to locate features, which represent distinguishable points that are present in most of the images considered. For example, for facial landmarks, the location of the left pupil can be used. If the initial landmarks are not recognizable (for example, if the person has an eye patch), secondary landmarks can be used. Such a landmark identification process can be used for any such object. In some examples, a set of landmarks forms a shape. The shape can be represented as a vector using the coordinates of the points in the shape. One shape is aligned with another shape using a similarity transformation (allowing translation, scaling, and rotation) that minimizes the average Euclidean distance between the shape points. The average shape is the average of the aligned training shapes.

[0067] In some examples, a landmark search begins with an average shape aligned with the position and size of a face determined by a global face detector. Such a search then repeats the steps of suggesting tentative shapes by adjusting the positions of shape points by template matching of the image texture around each point, and then conforming the tentative shapes to a global shape model until convergence occurs. In some systems, individual template matches are unreliable, and the shape model pools the results of weak template matches to form a stronger overall classifier. The entire search is repeated at each level of the image pyramid from coarse resolution to fine resolution.

[0068] The transformation system can capture an image or video stream on a client device (e.g., client device 102) and perform complex image manipulations locally on the client device 102 while maintaining an appropriate user experience, computational time, and power consumption. Complex image manipulations can include size and shape changes, emotion transformations (e.g., changing a face from a frown to a smile), state transitions (e.g., aging an object, reducing apparent age, changing gender), style transfer, application of graphical elements, and any other suitable image or video manipulations enabled by a convolutional neural network that has been configured to execute efficiently on the client device 102.

[0069] In some examples, a computer animation model for transforming image data can be used by a system in which a user can capture an image or video stream of the user (e.g., a selfie) using a client device 102 having a neural network that operates as part of a messaging client 104 operating on the client device 102. A transformation system operating within the messaging client 104 determines the presence of a face in the image or video stream and provides a modification icon associated with the computer animation model to transform the image data, or the computer animation model can be presented in association with an interface described herein. The modification icon includes the following changes, which can be the basis for modifying the user's face in the image or video stream as part of the modification operation. Once the modification icon is selected, the transformation system initiates a process of transforming the user's image to reflect the selected modification icon (e.g., generating a smiley face on the user). Once the image or video stream is captured and the specified modification is selected, the modified image or video stream can be presented in a graphical user interface displayed on the client device 102. The transformation system can implement a complex convolutional neural network on a portion of the image or video stream to generate and apply the selected modification. That is, a user can capture an image or video stream and once a modification icon is selected, the modification result is presented in real time or near real time. In addition, when the video stream is being captured, the modification can be persistent and the selected modification icon remains toggled. Machine-taught neural networks can be used to implement such modifications.

[0070] The graphical user interface presenting the modifications performed by the transformation system can provide additional interactive options for the user. Such options can be based on the interface used to initiate content capture and selection of a particular computer animation model (e.g., initiated from a content creator user interface). In various examples, the modification can be persistent after the initial selection of the modification icon. The user can turn the modification on or off by tapping or otherwise selecting the face modified by the transformation system, and store it for later viewing or browsing to other areas of the imaging application. In the case of multiple faces modified by the transformation system, the user can globally turn the modification on or off by tapping or selecting a single face modified and displayed within the graphical user interface. In some examples, individual faces in a group of multiple faces can be modified individually, or such modifications can be individually switched by tapping or selecting an individual face or a series of individual faces displayed within the graphical user interface.

[0071] The story table 312 stores data about a collection of messages and associated images, videos, or audio data that are compiled into a collection (e.g., a story or gallery). The creation of a particular collection may be initiated by a particular user (e.g., each user for whom a record is maintained in the entity table 304). A user may create a "personal story" in the form of a collection of content that has been created and sent / broadcasted by the user. To this end, the user interface of the messaging client 104 may include user-selectable icons to enable the sending user to add specific content to his or her personal story.

[0072] The collection may also constitute a "live story" that is a collection of content from multiple users, the content being created manually, automatically, or using a combination of manual and automatic techniques. For example, a "live story" may constitute a curated stream of user-submitted content from different locations and events. Users whose client devices have location services enabled and who are at a common location event at a particular time may be presented with the option of contributing content to a particular live story, for example, via the user interface of the messaging client 104. Live stories may be identified to a user by the messaging client 104 based on his or her location. The end result is a "live story" told from a group perspective.

[0073] Another type of content collection is called a "location story," which enables users whose client devices 102 are located within a particular geographic location (e.g., on a college or university campus) to contribute to a particular collection. In some examples, contributions to location stories may require secondary authentication to verify that the end user belongs to a particular organization or other entity (e.g., is a student on a university campus).

[0074] As mentioned above, the video table 314 stores video data, which in one example is associated with the message for which a record is maintained within the message table 302. Similarly, the image table 316 stores image data, which is associated with the message whose message data is stored in the entity table 304. The entity table 304 can associate various enhancements from the enhancement table 310 with the various images and videos stored in the image table 316 and the video table 314.

[0075] The distributed real-time status update server data 318 stores instances, nodes, and routing tables or information used by the distributed real-time status update server 107. For example, the distributed real-time status update server data 318 stores FE instances, RTG instances, routing tables, proxy objects or nodes, device objects, and inter-region proxy instances or nodes.

[0076] Data communication architecture

[0077] Figure 4is a schematic diagram illustrating the structure of a message 400 according to some examples, the message 400 being generated by a messaging client 104 for transmission to another messaging client 104 or a messaging server 114. The content of a particular message 400 is used to populate a message table 302 stored in a database 120, which is accessible by a messaging server 114. Similarly, the content of the message 400 is stored in memory as "in-flight" or "in-flight" data of a client device 102 or an application server 112. The message 400 is shown to include the following example components:

[0078] Message identifier 402 : a unique identifier that identifies the message 400 .

[0079] Message text payload 404 : Text to be generated by a user via the user interface of the client device 102 and included in the message 400 .

[0080] Message image payload 406 : Image data captured by a camera component of the client device 102 or retrieved from a memory component of the client device 102 and included in the message 400 . Image data for a sent or received message 400 may be stored in the image table 316 .

[0081] Message video payload 408 : Video data captured by the camera component or retrieved from the memory component of the client device 102 and included in the message 400 . The video data for a sent or received message 400 may be stored in the video table 314 .

[0082] Message audio payload 410 : audio data captured by a microphone or retrieved from a memory component of the client device 102 and included in the message 400 .

[0083] Message enhancement data 412: Enhancement data (e.g., filters, stickers, or other annotations or enhancements) representing enhancements to be applied to the message image payload 406, message video payload 408, or message audio payload 410 of the message 400. The enhancement data for a sent or received message 400 may be stored in the enhancement table 310.

[0084] Message duration parameter 414: A parameter value indicating the amount of time in seconds that the content of the message (e.g., message image payload 406, message video payload 408, message audio payload 410) will be presented to or made accessible to the user via the messaging client 104.

[0085] Message geolocation parameters 416: Geolocation data (e.g., latitude and longitude coordinates) associated with the content payload of the message. Multiple message geolocation parameter 416 values ​​may be included in the payload, with each of these parameter values ​​being associated with a content item included in the content (e.g., a specific image within a message image payload 406 or a specific video within a message video payload 408).

[0086] Message story identifier 418: An identifier value that identifies one or more content collections (e.g., "stories" identified in story table 312) with which a particular content item in message image payload 406 of message 400 is associated. For example, an identifier value may be used to associate multiple images within message image payload 406 with each of multiple content collections.

[0087] Message tags 420: Each message 400 may be tagged with a plurality of tags, each of which indicates the subject of the content included in the message payload. For example, where a particular image included in the message image payload 406 depicts an animal (e.g., a lion), a tag value indicating the relevant animal may be included in the message tags 420. The tag values ​​may be manually generated based on user input, or may be automatically generated using, for example, image recognition.

[0088] • Message sender identifier 422: An identifier (eg, a messaging system identifier, an email address, or a device identifier) ​​that indicates the user of the client device 102 on which the message 400 was generated and from which the message 400 was sent.

[0089] • Message recipient identifier 424: Indicates an identifier (eg, a messaging system identifier, an email address, or a device identifier) ​​of the user of the client device 102 to which the message 400 is addressed.

[0090] The content (e.g., value) of the various components of message 400 may be pointers to locations in a table in which the content data values ​​are stored. For example, the image value in message image payload 406 may be a pointer to a location in image table 316 (or the address of a location in image table 316). Similarly, the value in message video payload 408 may point to data stored in video table 314, the value stored in message enhancement data 412 may point to data stored in enhancement table 310, the value stored in message story identifier 418 may point to data stored in story table 312, and the values ​​stored in message sender identifier 422 and message recipient identifier 424 may point to user records stored in entity table 304.

[0091] Figure 5It is a diagrammatic representation of a distributed real-time status update server 107 according to some examples. The distributed real-time status update server 107 includes one or more FE instances 510, one or more RTG instances 520, and an RTG configuration diagram 530. A given client device 102 establishes a gRPC connection with a first FE instance 510. In one example, when a given client device 102 opens a chat interface or a map interface of a messaging client 104, the given client device 102 establishes a gRPC connection with a first FE instance 510. If the given client device 102 has not been previously connected to the distributed real-time status update server 107, or has been connected for more than a threshold time before the current time, a new FE instance 510 is created. The first FE instance 510 generates a proxy object 512 for communicating with the given client device 102. The proxy object 512 sends messages from the given client device 102 to the corresponding device object stored on the first RTG instance 520.

[0092] The FE instance 510 periodically obtains routing information from the RTG configuration map 530, and obtains routing information from the RTG configuration map 530 when the FE instance 510 is generated. The routing information identifies the storage location of multiple device objects on multiple RTG instances 520. The FE instance 510 determines whether the identifier of the client device 102 is included in the obtained routing information. If so, the FE instance 510 receives an update from the client device 102 via the proxy object 512, and sends the update to the device object associated with the client device 102 stored on the RTG instance. For example, the FE instance 510 determines that the first device object 522 corresponds to the identifier of the client device 102, and determines that the first device object 522 is maintained or stored on the first RTG instance 520. In this case, the proxy object 512 on the FE instance 510 sends an update to the first device object 522 by sending a message directed to the first device object 522 to the first RTG instance 520. The first RTG instance 520 also sends the information stored in the first device object 522 back to the FE instance 510 connected to the client device 102 associated with the first device object 522. To this end, the first RTG instance 520 determines the unique random identifier assigned to the proxy object 512 from which the message was received, and stores the identifier in the routing information 529. The first RTG instance 520 uses the identifier stored in the routing information 529 to identify the proxy object 512 and the FE instance 510 currently connected to the client device 102. The FE instance 510 provides the information received from the RTG instance 520 (the content of the device object 522) to the client device 102 via the proxy object 512, and the client device 102 then updates the graphical user interface (for example, updating the location of a friend on a map in real time).

[0093] The FE instance 510 may determine that the identifier of the client device 102 is missing or not included in the obtained routing information. In this case, the FE instance 510 instructs one of the RTG instances 520 to generate a new device object for the client device 102. The FE instance 510 then provides data from the client device 102 to the device object via the proxy object 512, or provides data from the device object to the client device 102.

[0094] Each RTG instance 520 stores one or more device objects. For example, the first RTG instance 520 stores a first device object 522 and a second device object 524. The second RTG instance stores a third device object 526. Each RTG instance 520 periodically obtains routing information 529, which identifies which device objects are stored in the corresponding RTG instance 520. The routing information 529 indicates the range of device identifiers processed by each corresponding RTG instance 520. The first device object 522 can process updates received from the client device 102 and identify one or more friends stored on the first device object 522. The first device object 522 generates a message including an update of the device object for transmission to the friend. The first message can be sent directly to the second device object 524 within the first RTG instance 520. The second message that needs to be delivered to the third device object 526 can be sent to the third device object 526 via one or more FE instances 510. That is, the third device object 526 can be on an RTG instance outside the RTG instance 520 storing the first device object. In this case, the RTG instance 520 selects a given FE instance 510 or a proxy object on one of the FE instances in a random or cyclic or round-robin manner. The RTG instance 520 sends a message with an identifier of the third device object 526 to the selected FE instance 510 or proxy object. The selected FE instance 510 receives the message and determines which of the RTG instances stores the third device object 526 based on the routing information 514 stored on the selected FE instance 510. Then, the selected FE instance 510 or proxy object sends the message to the identified or determined RTG instance storing the third device object 526 to update the information of the user associated with the first device object 522.

[0095] As an example, the first device object 522 may update the location of the client device 102. The location update may be shared with other friends of the user of the client device 102. The third device object 526 may store a list of friends and their corresponding locations. When the third device object 526 receives an update to the location from the first device object 522, the third device object 526 identifies the user associated with the first device object 522 and updates the current location stored for the user. The FE instance 510 may establish a connection with the friend's client device 102 using another proxy object stored in the FE instance 510 corresponding to the third device object 526. After the connection is established, the friend's location stored in the third device object 526 is provided to the friend's client device 102 to update the graphical user interface presented on the client device 102. In this way, the user can receive almost instant updates to the location of their friends on the map and see their friends moving actively on the map.

[0096] In some examples, when the FE instance 510 is initially generated and stored, a random identifier is assigned to the FE instance 510 (e.g., a proxy object stored on the FE instance 510). Each device object stored in the RTG instance is assigned an identifier corresponding to a user identifier associated with the device object. For example, the first device object 522 is associated with a user of the client device 102 and is assigned an identifier that matches a user identifier (e.g., a user name) of the user of the client device 102. In some cases, the FE instance communicates only with the RTG instance, without directly communicating or exchanging messages with each other. In some implementations, the first RTG instance 520 stores a random identifier of a proxy object 512 stored on the FE instance 510, which sends a message to the first RTG instance 520. When any device object in the first RTG instance 520 needs to send a message back to the proxy object 512, the first RTG instance 520 uses the stored random identifier to find the appropriate proxy object and routes the message to the appropriate proxy object, and then routes it to the corresponding client device 102.

[0097] The arrangement and interaction between FE instances 510 and RTG instances 520 makes it possible to have thousands of device objects in memory without having to perform complex synchronization operations. This also makes the system scalable and allows objects to reside anywhere geographically and on a variety of storage devices in a distributed manner.

[0098] In some examples, the grouped FE instances 510 and RTG instances 520 are divided into different geographical areas. For example, a first group of FE instances 510 and RTG instances 520 are stored in a first geographical area (e.g., the United States), while a second group of FE instances 510 and RTG instances 520 are stored in a second geographical area (e.g., Australia). Messages exchanged between the FE instances 510 and RTG instances 520 in the first group in the same geographical area are sent in real time. Messages exchanged between the FE instances 510 in the first group and the FE instances 510 in the second group are grouped and periodically sent in batches. In this way, the number of times messages are sent between different geographical areas is reduced, which makes better use of network resources and bandwidth.

[0099] Figure 6 It is a diagrammatic representation of a distributed real-time status update server 107 across different geographic regions according to some examples. Specifically, users are associated with the regions in which they are geographically located. For example, a first user in a first geographic region is associated with a device object in the first geographic region. A second user in a second geographic region is associated with a device object in the second geographic region. The mapping between a user and the region in which they are located and therefore between a user and the region in which their corresponding device object is located is stored in a globally replicated database table. The table is shared and replicated in various geographic regions. In some cases, when a user moves to a different geographic region, the mapping is updated periodically or in real time. In response to determining that the user has changed the geographic region, the device object stored for the user in the first geographic region is moved or copied to an RTG instance in the geographic region corresponding to the new user location. In some implementations, the device object is moved or copied, and the geographic region assigned to the user is updated immediately, or updated after the user has stayed in the new geographic region for a threshold number of days.

[0100] In some examples, a first device object 522 in a first geographic area may generate a message directed to a second device object in a second geographic area. Specifically, the first device object 522 searches a globally replicated database for a user identifier associated with the second device object to determine whether the second device object is in the first geographic area. If it is determined that the second device object is in the first geographic area, the first device object 522 sends the message to the RTG instance in which the second device object is stored. If it is determined that the second device object is in the second geographic area, the first device object 522 identifies a first inter-regional proxy node corresponding to the second area. The first inter-regional proxy node may be located in the first geographic area or any other suitable location. The first device object 522 sends a message with identifiers of the second user and the second device object to the identified inter-regional proxy node via the FE instance 510.

[0101] The first inter-regional proxy node collects and bundles multiple messages directed from the first geographic area and other geographic areas. Once the first inter-regional proxy node determines that a threshold time period has passed since the first inter-regional proxy node last sent a message to a second inter-regional proxy node located in the second geographic area, the first inter-regional proxy node sends the collected and bundled multiple messages to the second inter-regional proxy node. In some implementations, once the first inter-regional proxy node determines that a threshold number of collected messages has been reached, the first inter-regional proxy node sends the collected and bundled multiple messages to the second inter-regional proxy node. The second inter-regional proxy node located in the second geographic area receives the bundled message from the first inter-regional proxy node and distributes the message (e.g., randomly) to the FE instance in the second geographic area. Specifically, the FE instance in the second geographic area receives a message directed to a second device object from the second inter-regional proxy node, and identifies an RTG instance in the second geographic area that includes the second device object. The FE instance provides a message to the second device object via the identified RTG instance.

[0102] Figure 7 is a diagrammatic representation of an RTG instance 520 according to some examples. The RTG instance 520 is combined with Figure 5 and Figure 6 520 , an example of one of the many RTG instances discussed. As an example, the RTG instance 520 includes routing information 529, a device object 522, an input channel 730, a goroutine 720 (or coroutine), and can communicate with a remote database 710. In one example, the device object 522 stores various information for a given user, such as the user's last known location, the user's friend list, the user's friend group, the last update time of the device object 522, the status and location information of each friend in the friend list or friend group.

[0103] The device object 522 receives messages via the input channel 730. For example, the device object 522 receives a location update for a user associated with the device object 522 or a location update for a friend of the user stored in a friend list. The device object 522 receives an online status update for one or more friends of the user listed in the device object 522. The goroutine 720 (or coroutine) can manipulate the device data stored in the device object 522. It does so by looping through the messages received via the input channel 730 and taking actions on the messages based on the message type. At some point, the goroutine 720 may decide that the device object is no longer needed. When this happens, the device object is deleted from the graph and the goroutine 720 is terminated. The goroutine 720 copies any relevant information stored in the device object 522 to the remote database 710. The information can be associated with a user identifier corresponding to the device object 522, and when a new device object 522 is created for the same user at some later time, the information is retrieved and copied to the new device object 522.

[0104] Whenever a streaming gRPC connection is established between a client device 102 and a distributed real-time status update server, for example using a proxy object in a FE instance 510, a device object 522 for a given user is generated. In some cases, a device object 522 for a given user is generated when a friend of the user sends a message directed to the given user. At startup, when the device object 522 is initially created, the device object 522 retrieves the initial friend list from the messaging server 114. The device object 522 then sends a message to each device object corresponding to a friend in the initial friend list. That is, the device object 522 identifies the user identifier of each friend in the list, and sends a message to each device object associated with a corresponding one of the identified user identifiers. In some cases, the device object 522 sends messages only to those friends who are determined to be in an active state (e.g., users who are currently online). The message announces that the device object 522 has just become online.

[0105] After sending a message announcing that the device object 522 has become online, goroutine 720 begins processing messages received on the input channel 730 in a specific loop. The loop includes processing the input message (if any) according to the type of the input message. That is, different functions can be performed for different types of messages. Then, goroutine 720 checks expired friend locations and deletes these locations if any locations are expired. goroutine 720 checks whether the state needs to be persisted. If the state is dirty and if the state has not lasted for a threshold period of time, the state is persisted. Next, goroutine 720 checks the preferences that need to be refreshed periodically. Optionally, goroutine 720 next accesses a second remote source of user location (e.g., a third-party source) to pull or determine the current location of the user associated with the device object 522. If goroutine 720 determines that the pulled location from the second remote source is different by a certain factor (e.g., more than a certain distance relative to the current location stored for the user in the device object 522), goroutine 720 updates the current location based on the location retrieved from the second remote source. Finally, goroutine 720 checks for inactivity of the client device 102 or the user associated with the device object 522. If the device or user has been inactive for a threshold period of time (e.g., 8 hours), such as if no messages have been received on the input channel 730 of the device object 522, the information from the device object 522 is moved to the remote database 710 and associated with the identifier of the user corresponding to the device object 522. The device object 522 is then deleted and goroutine 720 is terminated. If the device or user has not been inactive for a threshold period of time, goroutine 720 restarts the loop including processing input messages on the input channel 730.

[0106] Fig. 8A is a diagrammatic representation of the message exchanges between device objects of a distributed real-time status update server according to some examples. Fig. 8A As shown, a first user 810 (eg, a first client device 102 of the first user 810 ) sends an update (eg, a status update or a location update) to a first device object 830 associated with the first user 810 via a streaming gRPC connection. Fig. 8A The links between the nodes shown show the direction of data flow, and the thickness of the links shows the type of connection (eg, real-time data exchange or rate-limited data exchange).

[0107] In one example, the first device object 830 receives a location update from the first user 810. In response, the first device object 830 pushes the location update to the second device object 832 corresponding to the second user 820. The second user 820 also provides the location update to the second device object 832 associated with the second user 820. The second device object 832 also pushes the location update from the second user 820 to the first device object 830 via the real-time link 840. Once the first device object 830 receives the update to the location from the second device object 832, the first device object 830 identifies the entry in the first device object 830 corresponding to the second user 820 and updates the location stored in the entry with the updated location received from the second device object 832.

[0108] The first device object 830 also pushes the location update to a third device object 834 corresponding to the third user. The third user may not send the location update back to the first user, and in this case, the third device object 834 only receives the location update from the first device object 830 without sending the location information back to the first device object 830.

[0109] In some implementations, location updates are exchanged in real time between the first device object 830 and the second device object 832. That is, when the first device object 830 receives a location update from the first user 810, the first device object 830 immediately sends the update to the second device object 832. This can be performed in response to the first device object 830 determining that the second user 820 associated with the second device object 832 has a status indicating that the second user 820 needs real-time location information. For example, if the second user 820 is currently viewing an interactive map showing the location of the second user's 820 friends, this status is sent and stored in the friend's entry in the first device object 830. The first device object 830, based on determining that the status of the second user 820 is viewing an interactive map showing the friend's location, sends the location update received from the first user 810 to the second device object 832 in real time when the update is received. In addition, the second device object 832 determines that the first user 810 is also viewing the interactive map of the friend's location, and therefore can share the location update received from the second user 820 with the first device object 830 in real time.

[0110] The first device object 830 may store a status of a third user associated with the third device object 834, indicating that the third user currently has the messaging client 104 open but is not viewing an interactive map showing the location of friends. In this case, the first device object 830 may periodically send or share with the third device object 834 the location updates received from the first user 810 over the rate-limited link 841, rather than in real time. That is, the first device object 830 may send updates to the location of the first user 810 to the third device object 834 only if the location of the first user 810 satisfies a first condition (e.g., changes by more than a first threshold amount, or if the location was last sent to the third device object 834 more than a first threshold amount of time ago (e.g., the time elapsed since the last update of the location was sent to the third device object 834 exceeds the first threshold)). In this way, simultaneously, the location updates of the first user 810 may be sent to some users in real time, but to other users periodically or not at all.

[0111] The first device object 830 may store a status of a fourth user associated with the fourth device object 836, indicating that the fourth user is currently inactive. In this case, the first device object 830 may periodically send or share with the fourth device object 836 the location updates received from the first user 810 via the further rate-constrained link 842, rather than in real time, or may not share the location at all. The rate of the further rate-constrained link 842 is more restricted than the rate-constrained link 841. When the first condition is met, the rate-constrained link 841 may allow the location to be exchanged or sent to the third device object 834. When the second condition is met and the first condition is not met, the further rate-constrained link 842 may allow the location to be exchanged or sent to the fourth device object 836. That is, the first device object 830 may send updates of the location of the first user 810 to the fourth device object 836 only if the location of the first user 810 satisfies the second condition (e.g., changes by more than a second threshold amount, or if the location was last sent to the fourth device object 836 before more than a second threshold amount of time (e.g., the time that has elapsed since the last update of the location was sent to the fourth device object 836 exceeds the second threshold)). Specifically, location updates shared with the fourth device object 836 may be further limited relative to those shared with the third device object 834 because the fourth user does not need to view the location information of the first user 810. The second threshold amount for limiting updates sent to the fourth device object 836 is greater than the first threshold amount for limiting updates sent to the third device object 834.

[0112] Figure 8Bis a diagrammatic representation of the message exchanges between device objects of a distributed real-time status update server according to some examples. Figure 8B As shown, an Online Update (OU) represents a message sent between devices to announce that the device is online and then periodically indicate that the device object is still online. When the client device 102 becomes online, an Online Update is sent from the client device 102 to the device object. When the client device 102 sends a location update, a Location Update (LU) is sent to the device object. When a user shares a location with his or her friends, a Friend Location Update (FLU) is sent between device objects. When a device object wants all other device objects to delete its location, such as when a given user enters ghost mode and does not want its location to be known, a Delete Location Update (DLU) is sent. A Preference Update (PU) is sent when a user's preferences are updated, so that the device object retrieves and updates the stored preferences. A Message Received Update (MRU) is sent to confirm message reception between device objects.

[0113] Figure 8B An example message exchange is provided when a device object initially becomes online. When a device object (e.g., device object A) becomes online and is first initialized, the device object sends an online update (OU) to all of its friends (e.g., device objects B, C, and D, which are friends of device object A). Each friend device object B, C, and D responds in one of three ways. For example, the first friend object B responds with a FLU that indicates the location of the friend stored in the first friend object B. This occurs when the device object receiving the OU shares its location with the device object of the user from whom the OU message was received. Device object A then stores the location received in the FLU message from the first friend object B in the entry for the first friend in device object A.

[0114] The second friend object C responds to the OU message received from the device object A with an online update (OU). This occurs when the device object receiving the OU does not share a location with the device object A from which the OU message was received, but is online (e.g., has a client device 102 physically connected to the second friend object C via a corresponding FE instance 510) and is interested in receiving location updates from the user associated with the device object A. That is, the second friend may desire to see or know the location of the user associated with the device object A, but may not be interested in sharing their location with the user associated with the device object A.

[0115] The third friend object D responds to the OU message received from the device object A with an MRU message. This occurs when the device object receiving the OU does not share a location with the device object A from which the OU message was received and is not interested in receiving location updates from the user associated with the device object A, such as because the third friend object D is offline (e.g., does not have a client device 102 physically connected to the third friend object D via the corresponding FE instance 510). That is, the third friend may not desire to see or know the location of the user associated with the device object A, and may not be interested in sharing their location with the user associated with the device object A.

[0116] Figure 8C is a diagrammatic representation of the message exchanges between device objects of a distributed real-time status update server according to some examples. Figure 8C As shown in FIG. 1 , in step 1: device object A initially becomes online and sends an OU message to all friends listed in device object A. In step 2: to process each OU message received from device object A, a new device object (e.g., an offline device object represented by a dashed circle) is created for each friend (B, C, D, and E). Since no friends are online or have a newly shared location, each of device objects B, C, D, and E responds to the OU message received from device object A with an MRU message.

[0117] In step 3: a given friend has a client device 102 that establishes a connection to device object C via FE instance 510. This causes device object C to become online - device object C switches from an offline device object to an online device object (represented by a solid circle). In response, device object C sends an OU message to all friends of device object C, including device object A. Device object A responds to the OU message received from device object C with a FLU message to indicate to device object C the current location of the user stored in device object A. In step 4: device objects A and C establish a connection based on the fact that they are both online and share locations with each other. They continue to exchange FLU messages to provide location updates to each other. At the same time, other device objects B, D, and E are deleted after a period of inactivity or after a threshold time period has passed since an event associated with the device object occurred. For example, if a threshold amount of time has passed since device object B was created, and the corresponding client device 102 has not established a connection with FE instance 510 to bring the device object online within the threshold amount of time, then device object B is deleted. As another example, if a threshold amount of time has passed since device object C was created and device object C has not received further messages (eg, from other users or from device object A), then device object C is deleted.

[0118] In some examples, the offline device object stores only the location information or location sharing preferences of the corresponding user. The online device object includes additional information not included in the offline object, such as a friend list of the first user, the connection status of the client device with the first device object, the connection status of the client device of the friend with the corresponding device object, the last time of each friend's activity, and the execution thread that provides the location estimate of the first user and the estimated location of the friend.

[0119] In step 5: device object C goes offline (e.g., because the client device 102 that was initially connected to device object C via FE instance 510 has been disconnected for a threshold period of time). In response, device object C stops sending FLU messages to device object A, but device object A continues to send FLU messages to device object C for a threshold period of time. In step 6: device object A stops sending FLU messages to device object C because device object A has not received FLU messages from device object C for a threshold period of time. After device objects A and C stop processing messages for a period of time (e.g., device objects A and C do not receive messages from the other device object or they do not receive messages from the client device 102 associated with the respective device objects), device objects A and C are deleted from the graph.

[0120] Although the discussion is based on the shared location update Figure 8B and Figure 8C , but similar functionality applies to sharing any other type of update (e.g., status update). In this case, instead of sending FLU messages and LU messages, the devices exchange status update messages or preference update messages.

[0121] Fig. 9 is a diagrammatic representation of sending updates based on different friend list versions according to some examples. In some examples, for each location update (LU) message provided by a given client device 102 and sent to a corresponding device object, the client device 102 also provides a friend list version number. This ensures that the friend list used by the device object of the client device 102 to send location updates to other device objects reflects the most recent changes to the user's friend list. In this way, if a given user deletes a friend or temporarily desires to prevent the friend from receiving location updates from the user, the given user's device object does not continue to share location updates based on an outdated copy of the friend list or preferences.

[0122] For example, as discussed above, at startup, the device object 930 of a given user (user A) retrieves the current friend list 932 from the messaging server 114. This is the friend list that the device object 930 uses to send OU messages to friend device objects. At this point, the client device 102 associated with the device object 930 also has the same friend list, so both share the same version number (e.g., 1000). At some later point, user A updates the friend list (e.g., unfriends the people on the friend list). In response, the client device 102 updates (increments) the version number of the friend list and creates a friend list 910 with a new version number. The update is sent to the messaging server 114. Shortly thereafter, the client device 102 sends a location update 920 to the distributed real-time status update server 107, and specifically to the device object 930. The location update 920 includes the new version number of the friend list. The device object 930 compares the version number received from the location update with the current version number stored on the device object 930. In response to determining that the version number stored on device object 930 is different from the version number of the friend list received in the location update, device object 930 temporarily blocks further location updates (FLU messages) from being sent to the friend device objects (B, C, D, and E).

[0123] The device object 930 communicates with the messaging server 114 or the client device 102 to obtain the latest friend list corresponding to the version number received in the location update. This synchronizes the friend list stored on the device object 930 with the friend list stored or recently changed by the user of the client device 102 corresponding to the device object 930. Once the friend list is updated, the device object 930 continues to send FLU messages with location updates to the friends on the updated friend list.

[0124] Fig. 10A 1 is a flow chart illustrating an example operation of the distributed real-time status update server 107 according to an example when performing the process 1000. The process 1000 may be implemented by computer-readable instructions for execution by one or more processors, so that the operations of the process 1000 may be performed in part or in whole by functional components of the distributed real-time status update server 107; therefore, reference is made to Fig. 10A Process 1000 is described as an example. However, in other examples, at least some of the operations of process 1000 may be deployed on various other hardware configurations, such as on application server 112. The operations in process 1000 may be performed in any order, in parallel, or may be skipped and omitted entirely.

[0125] At operation 1001, the distributed real-time status update server 107 stores a front end (FE) instance and a plurality of real-time graph (RTG) instances, each of the plurality of RTG instances including a plurality of device objects, and the FE instance is configured to communicate with a client device associated with a first user. For example, the distributed real-time status update server 107 stores one or more FE instances 510 and one or more RTG instances 520. The FE instance 510 includes a proxy object configured to communicate with the client device 102, and the RTG instance includes one or more device objects (e.g., an online device object or an offline device object).

[0126] At operation 1002 , the distributed real-time status update server 107 establishes a bidirectional streaming remote procedure call (RPC) connection between the FE instance and multiple RTG instances. For example, the FE instance 510 establishes an RPC connection with one or more RTG instances 520 .

[0127] At operation 1003 , the distributed real-time status update server 107 receives status updates from the client device via the FE instance. For example, the proxy object 512 of the FE instance 510 receives location updates from the client device 102 via a real-time streaming connection with the client device 102 .

[0128] At operation 1004, the distributed real-time status update server 107 determines through the FE instance that the first device object corresponding to the client device is stored on the first RTG instance among the multiple RTG instances. For example, the FE instance 510 identifies the device object 522 and which one of the RTG instances 520 stores the identified device object 522, for example, by accessing the routing information 514.

[0129] At operation 1005, the distributed real-time status update server 107 sends a first message including a status update from the FE instance to the first RTG instance to update the first device object. For example, the FE instance 510 sends the location update received from the client device 102 to the device object 522 in the identified RTG instance 520 via the gRPC connection.

[0130] Fig. 10B 1 is a flowchart illustrating an example operation of the distributed real-time status update server 107 according to an example when performing process 1001. Process 1001 can be implemented in computer-readable instructions for execution by one or more processors, so that the operations of process 1001 can be performed in part or in whole by functional components of the distributed real-time status update server 107; therefore, the following reference is made to Fig. 10BProcess 1001 is described as an example. However, in other examples, at least some of the operations of process 1001 may be deployed on various other hardware configurations, such as on application server 112. The operations in process 1001 may be performed in any order, in parallel, or may be skipped and omitted entirely.

[0131] At operation 1011, the distributed real-time status update server 107 stores one or more front-end (FE) instances and multiple real-time graph (RTG) instances. For example, the distributed real-time status update server 107 stores one or more FE instances 510 and one or more RTG instances 520. The FE instance 510 includes a proxy object configured to communicate with the client device 102, and the RTG instance includes one or more device objects (e.g., online device objects or offline device objects).

[0132] At operation 1012, the distributed real-time status update server 107 receives updates from a first client device of a given user via a first online device object associated with the given user. Figure 8C As shown in step 1 of , online device object A receives a location update from a FE instance 510 coupled to a first client device of a given user.

[0133] At operation 1013, the distributed real-time status update server 107 generates a message including an update for sending to a plurality of friends of a given user by the first online device object. Figure 8C As shown in steps 1 and 2 of , an online device object A sends an OU message to the device objects of friends of a given user (such as device objects B, C, D, and E).

[0134] At operation 1014, the distributed real-time status update server 107 stores an offline device object for a first friend among the plurality of friends and a second online device object for a second friend among the plurality of friends on a given instance among the plurality of RTG instances. Figure 8C As shown in step 3, the distributed real-time status update server 107 stores the offline device object B (shown by a dotted circle) and the online device object C (shown by a solid circle).

[0135] At operation 1015, the distributed real-time status update server 107 sends a message including an update to the offline device object of the first friend and the second online device object of the second friend via the first online device object. Figure 8C As shown in steps 1 to 3, the online device object A sends an OU message to the offline device object B (shown by a dotted circle) and the online device object C (shown by a solid circle).

[0136] Fig. 10C1 is a flowchart illustrating an example operation of the distributed real-time status update server 107 according to an example when executing process 1002. Process 1002 can be implemented by computer-readable instructions for execution by one or more processors, so that the operations of process 1002 can be performed in part or in whole by functional components of the distributed real-time status update server 107; therefore, the following reference is made to Fig. 10C Process 1002 is described as an example. However, in other examples, at least some of the operations of process 1002 may be deployed on various other hardware configurations, such as on application server 112. The operations in process 1002 may be performed in any order, in parallel, or may be skipped and omitted entirely.

[0137] At operation 1021, the distributed real-time status update server 107 stores a plurality of real-time graph (RTG) instances including a plurality of device objects. For example, the distributed real-time status update server 107 stores one or more FE instances 510 and one or more RTG instances 520. The FE instance 510 includes a proxy object configured to communicate with the client device 102, and the RTG instance includes one or more device objects (e.g., an online device object or an offline device object).

[0138] At operation 1022, the distributed real-time status update server 107 receives a status update from a client device associated with the first user via a first device object in the plurality of device objects. For example, the proxy object 512 of the FE instance 510 receives a location update from the client device 102 associated with the first user 810 via a real-time streaming connection with the client device 102. The proxy object 512 provides the location update to the first device object 830 associated with the first user 810.

[0139] At operation 1023, the distributed real-time status update server 107 sends a first message including a status update to a second device object associated with a second user via a real-time link by the first device object 830. For example, the first device object 830 sends a location update to the second device object 832 associated with the second user 820 via the real-time link 840.

[0140] At operation 1024, the distributed real-time status update server 107 sends a second message including a status update to a third device object associated with a third user via a first rate-constrained link by the first device object. For example, the first device object 830 sends a location update to a third device object 834 associated with a third user via a rate-constrained link 841.

[0141] Fig. 10D1 is a flowchart illustrating an example operation of the distributed real-time status update server 107 according to an example when executing process 1003. Process 1003 can be implemented by computer-readable instructions for execution by one or more processors, so that the operations of process 1003 can be performed in part or in whole by the functional components of the distributed real-time status update server 107; therefore, the following reference is made to Fig. 10D Process 1003 is described as an example. However, in other examples, at least some of the operations of process 1003 may be deployed on various other hardware configurations, such as on application server 112. The operations in process 1003 may be performed in any order, in parallel, or may be skipped and omitted entirely.

[0142] At operation 1031, the distributed real-time status update server 107 stores multiple real-time graph (RTG) instances including multiple device objects, including a first device object associated with a given user. For example, the distributed real-time status update server 107 stores one or more FE instances 510 and one or more RTG instances 520. The FE instance 510 includes a proxy object configured to communicate with the client device 102, and the RTG instance includes one or more device objects (e.g., online device objects or offline device objects).

[0143] At operation 1032, the distributed real-time status update server 107 receives a given user's friend list with a first version identifier through the first device object. For example, the device object 930 receives the user A's friend list from the messaging server 114. The friend list has a version number of 1000.

[0144] At operation 1033, the distributed real-time status update server 107 receives an update from a client device associated with a given user via a first device object, the update including a friend list version identifier. For example, the device object 930 receives a location update from a client device 102 associated with user A. The location update includes a version identifier of a friend list locally accessed or stored by the client device 102. The version identifier may have a version number 1001. As an example, the client device 102 may have unfriended a given friend or changed parameters to prevent sharing location with a given friend on the friend list. In response, the client device 102 may update the friend list on the messaging server 114 and increment the version identifier of the friend list.

[0145] At operation 1034, the distributed real-time status update server 107 determines that the first version identifier of the friend list in the first device object does not match the friend list version identifier in the update. For example, the device object 930 compares the version identifier stored by the device object 930 with the version identifier received in the update to ensure that the device object 930 is operating on the latest version of the friend list. The device object 930 performs this comparison before sending further updates to any device objects of the given user's friends.

[0146] At operation 1035, the distributed real-time status update server 107 synchronizes the friend list in the first device object before sending one or more messages including the update to other device objects in the plurality of device objects. For example, in response to the device object 930 determining that the current version identifier (e.g., 1000) does not match the version identifier (e.g., 1001) received in the update, the device object 930 accesses the friend list from the messaging server 114 or the client device 102 before sending further updates to the device objects associated with the given user's friends.

[0147] Machine Architecture

[0148] Fig.111100 within which instructions 1108 (e.g., software, programs, applications, applet, app, or other executable code) may be executed for causing the machine 1100 to perform any one or more of the methods discussed herein. For example, the instructions 1108 may cause the machine 1100 to perform any one or more of the methods described herein. The instructions 1108 convert a general-purpose, non-programmed machine 1100 into a specific machine 1100 that is programmed to perform the functions described and illustrated in the manner described. The machine 1100 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1100 may operate as a server machine or a client machine in a server-client network environment or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1100 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing instructions 1108 specifying actions to be taken by the machine 1100. In addition, although only a single machine 1100 is shown, the term "machine" should also be taken to include a collection of machines that individually or jointly execute instructions 1108 to perform any one or more of the methods discussed herein. For example, the machine 1100 may include any of the client device 102 or a number of server devices that form part of the messaging server system 108. In some examples, the machine 1100 may also include both a client and server system, wherein certain operations of a particular method or algorithm are performed on the server side and certain operations of a particular method or algorithm are performed on the client side.

[0149] The machine 1100 may include a processor 1102, a memory 1104, and an input / output (I / O) component 1138 that may be configured to communicate with each other via a bus 1140. In an example, the processor 1102 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 1106 that executes instructions 1108 and a processor 1110. The term "processor" is intended to include a multi-core processor that may include two or more independent processors (sometimes referred to as "cores") that may execute instructions simultaneously. Although Fig.11 A multiprocessor 1102 is shown, but the machine 1100 may include a single processor with a single core, a single processor with multiple cores (eg, a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0150] The memory 1104 includes a main memory 1112, a static memory 1114, and a storage unit 1116, all of which are accessible to the processor 1102 via the bus 1140. The main memory 1104, the static memory 1114, and the storage unit 1116 store instructions 1108 that implement any one or more of the methods or functions described herein. The instructions 1108 may also reside, completely or partially, within the main memory 1112, within the static memory 1114, within the machine-readable medium 1118 within the storage unit 1116, within at least one of the processors 1102 (e.g., within a cache memory of a processor), or within any suitable combination thereof during execution thereof by the machine 1100.

[0151] I / O components 1138 may include various components for receiving input, providing output, generating output, sending information, exchanging information, capturing measurements, etc. The specific I / O components 1138 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine will be less likely to include such a touch input device. It will be appreciated that I / O components 1138 may include Fig.1111 and 12. In various examples, the I / O components 1138 may include a user output component 1124 and a user input component 1126. The user output component 1124 may include a visual component (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), an acoustic component (e.g., a speaker), a tactile component (e.g., a vibration motor, a resistance mechanism), other signal generators, etc. The user input component 1126 may include an alphanumeric input component (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input component), a point-based input component (e.g., a mouse, a touch pad, a trackball, a joystick, a motion sensor, or other pointing instrument), a tactile input component (e.g., a physical button, a touch screen that provides the location and force of a touch or touch gesture, or other tactile input component), an audio input component (e.g., a microphone), etc.

[0152] In other examples, the I / O component 1138 may include a biometric component 1128, a motion component 1130, an environmental component 1132, or a positioning component 1134, as well as various other components. For example, the biometric component 1128 includes components for detecting expressions (e.g., hand expressions, facial expressions, vocal expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 1130 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, and a rotation sensor component (e.g., a gyroscope).

[0153] Environmental components 1132 include, for example, one or more cameras (with still image / photo and video capabilities), an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor that detects concentrations of hazardous gases for safety or measures pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment.

[0154] With respect to cameras, client device 102 can have a camera system including, for example, a front-facing camera on a front surface of client device 102 and a rear-facing camera on a rear surface of client device 102. The front-facing camera can, for example, be used to capture still images and videos of a user of client device 102 (e.g., a "selfie"), which can then be enhanced with the enhancement data (e.g., filters) described above. For example, the rear-facing camera can be used to capture still images and videos in a more traditional camera mode, which are similarly enhanced using the enhancement data. In addition to the front-facing camera and the rear-facing camera, client device 102 can also include a 360° camera for capturing 360° photos and videos.

[0155] Additionally, the camera system of the client device 102 may include dual rear cameras (e.g., a main camera and a depth sensing camera), or even triple, quad, or quintuple rear camera configurations on the front and back sides of the client device 102. For example, these multi-camera systems may include a wide-angle camera, an ultra-wide-angle camera, a telephoto camera, a macro camera, and a depth sensor.

[0156] The positioning component 1134 includes a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or a barometer that detects air pressure that can obtain the altitude), an orientation sensor component (e.g., a magnetometer), and the like.

[0157] A variety of technologies may be used to implement communications. The I / O components 1138 also include a communication component 1136 that is operable to couple the machine 1100 to the network 1120 or device 1122 via a corresponding coupling or connection. For example, the communication component 1136 may include a network interface component or another suitable device to interface with the network 1120. In other examples, the communication component 1136 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, Parts (e.g. Low power consumption), Components and other communication components that provide communication via other forms. Device 1122 can be any peripheral device among other machines or various peripheral devices (for example, a peripheral device coupled via USB).

[0158] In addition, the communication component 1136 can detect an identifier or include a component operable to detect an identifier. For example, the communication component 1136 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional bar codes such as universal product code (UPC) bar codes, multi-dimensional bar codes such as quick response (QR) codes, Aztec codes, data matrix, data symbol (Dataglyph), MaxiCode, PDF417, UltraCode, UCC RSS-2D bar codes, and other optical codes) or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be derived via the communication component 1136, such as the location of a geographic location via an Internet Protocol (IP), the location of a location via a wireless network, and the location of a location via a wireless network. The location of signal triangulation, the location of an NFC beacon signal via detection that can indicate a specific location, etc.

[0159] Various memories (e.g., main memory 1112, static memory 1114, and memory of processor 1102) and storage unit 1116 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 1108) when executed by processor 1102 cause various operations to implement the disclosed examples.

[0160] The instructions 1108 may be sent or received over the network 1120 via a network interface device (e.g., a network interface component included in the communications component 1136) using a transmission medium and using any of several well-known transmission protocols (e.g., the Hypertext Transfer Protocol (HTTP)). Similarly, the instructions 1108 may be sent or received via a coupling (e.g., a peer-to-peer coupling) to the device 1122 using a transmission medium.

[0161] Software Architecture

[0162] Fig.1212 is a block diagram 1200 illustrating a software architecture 1204 that can be installed on any one or more of the devices described herein. The software architecture 1204 is supported by hardware, such as a machine 1202, including a processor 1220, a memory 1226, and an I / O component 1238. In this example, the software architecture 1204 can be conceptualized as a stack of layers in which each layer provides specific functionality. The software architecture 1204 includes layers such as an operating system 1212, a library 1210, a framework 1208, and an application 1206. In operation, the application 1206 invokes an API call 1250 through the software stack and receives a message 1252 in response to the API call 1250.

[0163] The operating system 1212 manages hardware resources and provides public services. The operating system 1212 includes, for example, a kernel 1214, a service 1216, and a driver 1222. The kernel 1214 acts as an abstraction layer between the hardware layer and other software layers. For example, the kernel 1214 provides memory management, processor management (e.g., scheduling), component management, network and security settings, and other functions. The service 1216 can provide other public services to other software layers. The driver 1222 is responsible for controlling or interfacing with the underlying hardware. For example, the driver 1222 may include a display driver, a camera driver, or Low-power drivers, Flash drivers, Serial communication drivers (e.g., USB drivers), Drivers, audio drivers, power management drivers, etc.

[0164] The libraries 1210 provide a common low-level infrastructure used by the applications 1206. The libraries 1210 may include system libraries 1218 (e.g., C standard libraries) that provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the libraries 1210 may include API libraries 1224, such as media libraries (e.g., libraries for supporting presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), graphics libraries (e.g., OpenGL framework for presentation in two dimensions (2D) and three dimensions (3D) in graphics content on a display), database libraries (e.g., SQLite providing various relational database functions), web libraries (e.g., WebKit providing web browsing functions), etc. The library 1210 may also include a variety of other libraries 1228 to provide many other APIs to the application 1206 .

[0165] The framework 1208 provides a common high-level infrastructure used by the applications 1206. For example, the framework 1208 provides various graphical user interface (GUI) functions, high-level resource management, and high-level positioning services. The framework 1208 can provide a wide range of other APIs that can be used by the applications 1206, some of which can be specific to a specific operating system or platform.

[0166] In an example, applications 1206 may include a home application 1236, a contacts application 1230, a browser application 1232, a book reader application 1234, a location application 1242, a media application 1244, a messaging application 1246, a game application 1248, and a variety of other applications such as third-party applications 1240. Applications 1206 are programs that execute functions defined in the program. Various programming languages ​​may be used to create one or more of the applications 1206 constructed in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C or assembly language). In a specific example, third-party applications 1240 (e.g., applications created by an entity other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) can be used to develop applications on platforms such as IOS TM ANDROID TM , Mobile software running on the mobile operating system of the phone or other mobile operating systems. In this example, the third-party application 1240 can call the API call 1250 provided by the operating system 1212 to facilitate the functions described in this article.

[0167] Glossary

[0168] "Carrier signal" refers to any intangible medium that can store, encode or carry instructions for execution by a machine and includes digital or analog communications signals or other intangible media to facilitate communication of such instructions. Instructions may be sent or received over a network using a transmission medium via a network interface device.

[0169] "Client Device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communications device that a user may use to access a network.

[0170] "Communications network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, A network, other type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rate for GSM evolution (EDGE) technology, including the third generation partnership project (3GPP) of 3G, fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standard, other data transmission technologies defined by various standard setting organizations, other long distance protocols, or other data transmission technologies.

[0171] "Component" means a device, physical entity, or logic having boundaries defined by function or subroutine calls, branch points, APIs, or other techniques that provide partitioning or modularization of specific processing or control functions. A component may be combined with other components via its interfaces to perform a machine process. A component may be a packaged functional hardware unit designed for use with other components, and may be part of a program that generally performs a specific one of the related functions.

[0172] Components may constitute software components (e.g., code embodied on a machine-readable medium) or hardware components. A "hardware component" is a tangible unit that is capable of performing certain operations and may be configured or arranged in some physical manner. In various examples, one or more computer systems (e.g., stand-alone computer systems, client computer systems, or server computer systems) or one or more hardware components of a computer system (e.g., a processor or group of processors) may be configured by software (e.g., an application or application portion) as a hardware component for performing certain operations described herein.

[0173] The hardware components may also be implemented mechanically, electronically, or in any suitable combination thereof. For example, the hardware components may include a dedicated circuit system or logic that is permanently configured to perform certain operations. The hardware component may be a dedicated processor such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware components may also include a programmable logic or circuit system that is temporarily configured to perform certain operations by software. For example, the hardware components may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware components become a specific machine (or a specific component of a machine) that is uniquely customized to perform the configured function, and are no longer a general-purpose processor. It will be understood that it may be decided to implement the hardware components mechanically, in a dedicated and permanently configured circuit, or in a temporarily configured circuit (e.g., configured by software) for cost and time considerations. Accordingly, the phrase "hardware component" (or "component implemented by hardware") should be understood to include a tangible entity, i.e., an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in some way or perform certain operations described herein.

[0174] Consider an example where a hardware component is temporarily configured (e.g., programmed), without configuring or instantiating each of the hardware components at any one time. For example, where the hardware component includes a general-purpose processor that is configured by software to become a special-purpose processor, the general-purpose processor can be configured as different special-purpose processors (e.g., including different hardware components) at different times. The software configures one or more specific processors accordingly, for example, to constitute a specific hardware component at one time, and to constitute different hardware components at different times.

[0175] Hardware components can provide information to other hardware components and receive information from other hardware components. Therefore, the described hardware components can be considered to be communicatively coupled. In the case of multiple hardware components simultaneously, communication can be achieved by signal transmission between or among two or more hardware components (e.g., by appropriate circuits and buses). In an embodiment in which multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessed by multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a memory device to which it is communicatively coupled. Then, other hardware components can access the memory device at a subsequent time to retrieve the stored output and process the stored output. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (e.g., the collection of information).

[0176] The various operations of the example methods described herein may be performed at least in part by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform related operations. Whether temporarily configured or permanently configured, such a processor may constitute a processor-implemented component that operates to perform one or more operations or functions described herein. As used herein, a "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the method described herein may be implemented at least in part by a processor, wherein a specific one or more processors are examples of hardware. For example, at least some operations of the method may be performed by one or more processors 1102 or a processor-implemented component. In addition, the one or more processors may also be operated to support the execution of related operations in a "cloud computing" environment or to operate as "software as a service" (SaaS). For example, at least some operations in the operation may be performed by a computer group (as an example of a machine including a processor), wherein these operations may be accessed via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., API). The execution of certain operations may be distributed between processors, not residing only in a single machine, but deployed across multiple machines. In some examples, the processor or processor-implemented components may be located in a single geographic location (e.g., in a home environment, an office environment, or a server farm). In other examples, the processor or processor-implemented components may be distributed across several geographic locations.

[0177] "Computer-readable storage media" refers to both machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium", "computer-readable medium" and "device-readable medium" mean the same thing and can be used interchangeably in this disclosure.

[0178] "Ephemeral messages" are messages that are accessible for a limited duration in time. Ephemeral messages can be text, images, videos, etc. The access time for ephemeral messages can be set by the message sender. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technique, the message is ephemeral.

[0179] “Machine storage media” refers to a single or multiple storage devices and media (e.g., centralized or distributed databases, and associated caches and servers) that store executable instructions, routines, and data. Thus, the term should be considered to include, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGAs, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms “machine storage media,” “device storage media,” and “computer storage media” mean the same thing and are used interchangeably in this disclosure. The terms “machine storage media,” “computer storage media,” and “device storage media” expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are encompassed by the term “signal media.”

[0180] “Non-transitory computer-readable storage medium” refers to tangible media capable of storing, encoding, or carrying instructions to be executed by a machine.

[0181] "Signal medium" refers to any intangible medium that is capable of storing, encoding or carrying instructions executed by a machine, and includes digital or analog communication signals or other intangible media to facilitate the communication of software or data. The term "signal medium" should be deemed to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" refer to the same thing and may be used interchangeably in this disclosure.

[0182] Changes and modifications may be made to the disclosed examples without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure as expressed in the appended claims.

Claims

1. A method comprising: storing a plurality of real-time graph (RTG) instances including a plurality of device objects on a distributed storage system; receiving, by a first device object of the plurality of device objects, a status update from a client device associated with a first user; In response to the first device object determining that the second user has a current status indicating a need for a real-time status update, sending, by the first device object, a first message including the status update to a second device object associated with the second user via a real-time link; as well as In response to the first device object determining that the third user does not have a current status indicating that a real-time status update is required, sending, by the first device object, a second message including the status update to a third device object associated with the third user over a first rate-constrained link, In response to determining that the status update satisfies a first condition, the first rate-constrained link allows the status update to be sent to the third device object.

2. The method according to claim 1, further comprising: In response to determining that the second user's current status indicates that the second user is currently viewing an interactive map of avatars including the first user's avatar, the first message is sent over the real-time link.

3. The method according to claim 2, further comprising: In response to determining that the third user's current status indicates that the third user has a messaging application open and is not currently viewing the interactive map, sending the second message over the first rate-constrained link.

4. The method according to claim 1, further comprising: A third message including the status update is sent by the first device object to a fourth device object associated with a fourth user over a second rate constrained link, the second rate constrained link being further rate constrained relative to the first rate constrained link.

5. The method according to claim 4, further comprising: In response to determining that the current status of the fourth user indicates that the fourth user is currently inactive on the messaging application, sending the third message over the second rate-constrained link.

6. The method according to claim 5, wherein: In response to determining that the status update satisfies a second condition, the second rate-constrained link enables the status update to be sent to the fourth device object.

7. The method according to claim 6, wherein: The first condition includes a position indicated by the status update having changed by more than a first threshold amount relative to a previously stored position, and wherein the second condition includes a position indicated by the status update having changed by more than a second threshold amount relative to a previously stored position, the second threshold amount being greater than the first threshold amount.

8. The method according to claim 6, wherein: The first condition comprises a first elapsed time since the location indicated by the status update was last sent to the device object, and wherein the second condition comprises a second elapsed time since the location indicated by the status update was last sent to the device object.

9. The method of claim 1, further comprising storing a front-end (FE) instance on the distributed storage system, the FE instance being configured to communicate with the first client device.

10. The method of claim 9, further comprising establishing a bidirectional streaming remote procedure call (RPC) connection between the FE instance and the plurality of RTG instances.

11. The method according to claim 9, wherein: The FE instance includes a proxy object and routing information, a first portion of the routing information indicating that a first RTG instance includes a first subset of the plurality of device objects, and a second RTG instance of the plurality of RTG instances includes a second subset of the plurality of device objects.

12. The method according to claim 1, wherein: The first device object and the second device object are stored on a first RTG instance among the multiple RTG instances, further comprising: The first message is sent from the first device object to the second device object within the first RTG instance.

13. The method according to claim 1, wherein: The first device object is stored on a first RTG instance among the multiple RTG instances, and the second device object is stored on a second RTG instance among the multiple RTG instances, further comprising: The first message is sent by the first RTG instance to a given front end (FE) instance, wherein the given FE instance routes the first message to the second RTG instance to update the second device object.

14. The method according to claim 1, wherein: The status update includes location information or location sharing preference.

15. The method according to claim 1, wherein: The second device object comprises an offline device object, and the third device object comprises an online device object.

16. A system comprising: A processor configured to perform operations comprising: storing a plurality of real-time graph (RTG) instances including a plurality of device objects on a distributed storage system; receiving, by a first device object of the plurality of device objects, a status update from a client device associated with a first user; In response to the first device object determining that the second user has a current status indicating a need for a real-time status update, sending, by the first device object, a first message including the status update to a second device object associated with the second user over a real-time link; and In response to the first device object determining that the third user does not have a current status indicating that a real-time status update is required, sending, by the first device object, a second message including the status update to a third device object associated with the third user over a first rate-constrained link, In response to determining that the status update satisfies a first condition, the first rate-constrained link allows the status update to be sent to the third device object.

17. The system of claim 16, wherein: The operations also include: In response to determining that the second user's current status indicates that the second user is currently viewing an interactive map of avatars including the first user's avatar, the first message is sent over the real-time link.

18. A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising: storing a plurality of real-time graph (RTG) instances including a plurality of device objects on a distributed storage system; receiving, by a first device object of the plurality of device objects, a status update from a client device associated with a first user; In response to the first device object determining that the second user has a current status indicating a need for a real-time status update, sending, by the first device object, a first message including the status update to a second device object associated with the second user via a real-time link; as well as In response to the first device object determining that the third user does not have a current status indicating that a real-time status update is required, sending, by the first device object, a second message including the status update to a third device object associated with the third user over a first rate-constrained link, In response to determining that the status update satisfies a first condition, the first rate-constrained link allows the status update to be sent to the third device object.

Images