Implementing shared augmented reality video calls with augmented reality data channels

Abstract

Systems, methods, client devices, and non-transitory computer-readable media are disclosed for implementing shared augmented reality video calls with an augmented reality (AR) data channel that facilitate sharing of AR elements and interactions with AR elements during a video call. For example, the disclosed systems can establish a video call between multiple client devices that includes a video (and audio) data channel and an AR data channel. Further, in one or more embodiments, the disclosed systems enable one of the multiple client devices to send AR data over the AR data channel to cause another client device to render an AR element on video captured by the other client device during the video call. Indeed, the disclosed systems can implement AR environments, AR effects, AR-based activities, and / or individual AR elements during a video call with the AR data channel.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of priority to U.S. non-provisional application No. 17 / 650484, filed February 9, 2022, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] This disclosure generally relates to video calling systems. Background Technology

[0004] This disclosure generally relates to video conferencing systems. Video conferencing systems allow multiple users to communicate electronically via multiple computing devices (e.g., smartphones, laptops, tablets, desktop computers) using audio and video inputs (e.g., built-in digital cameras, digital webcams). Indeed, electronic communication via video conferencing and video conferencing has increased in recent years, enabling multiple users to communicate via computing devices to share both their video and audio with each other. However, conventional video conferencing systems are typically limited to non-interactive video conferencing, which simply and strictly allows multiple user devices to present and view the captured video across these user devices. Summary of the Invention

[0005] The embodiments of this disclosure provide a variety of advantages and / or address one or more of the foregoing or other problems in the art by utilizing a system, non-transitory computer-readable medium, and method for sharing augmented reality video calls using a dedicated augmented reality data channel. This facilitates the sharing and interaction with augmented reality elements during video calls. For example, the disclosed system can establish video calls between multiple client devices, including video (and audio) data channels and AR data channels. Furthermore, in one or more embodiments, the disclosed system enables one of the multiple client devices to transmit AR data (e.g., AR element identifiers (also referred to as AR identifiers), AR element information (also referred to as AR elements), logical data objects, object vectors, participant identifiers) via the AR data channel, so that another client device renders AR elements on video captured by that other client device.

[0006] For example, the disclosed system can enable shared AR environments (and / or AR effects) during video calls, allowing the sharing of AR environment scenes (and effects). In addition to AR environment scenes within a video call, the disclosed system can also facilitate the transition (and interaction with multiple AR objects) of AR objects (e.g., AR objects within an AR environment scene) between videos captured by multiple client devices participating in the video call. Furthermore, the disclosed system can utilize the transition (and interaction with AR objects) of AR objects between videos captured by multiple client devices to enable AR-based gaming between multiple participating client devices during a video call.

[0007] Additional features and advantages of one or more embodiments of this disclosure are outlined in the following description, which will be apparent in part from the description or may be learned by practice of these exemplary embodiments. Attached Figure Description

[0008] The specific embodiments are described with reference to the accompanying drawings, in which:

[0009] Figure 1 An example environment in which a shared AR video calling system, according to one or more implementations, may operate is shown.

[0010] Figure 2 An example is shown of a shared AR video call system according to one or more embodiments, which facilitates the establishment of a shared AR video call among multiple participant devices.

[0011] Figure 3 A flowchart is shown illustrating the establishment of an AR data channel during a shared AR video call system, and the transmission of AR data by multiple client devices, according to one or more embodiments.

[0012] Figure 4 A shared AR video call system according to one or more embodiments is shown, which establishes a video call with a shared AR environment scene between multiple client devices.

[0013] Figure 5 An AR video call system for sharing during a video call, according to one or more embodiments, is shown, enabling AR elements to be moved from one client device to another.

[0014] Figures 6A to 6D A shared AR video call system according to one or more embodiments is shown to enable AR-based activities between multiple client devices in a video call using an AR data channel.

[0015] Figure 7This paper illustrates a shared AR video call system according to one or more embodiments, which enables shared AR effects among multiple client devices during a video call.

[0016] Figure 8 A flowchart illustrating a series of actions for rendering AR elements within a video call, according to one or more embodiments, is shown.

[0017] Figure 9 A flowchart illustrating a series of actions for establishing a shared augmented reality video call, according to one or more embodiments, is shown.

[0018] Figure 10 A block diagram of an example computing device according to one or more embodiments is shown.

[0019] Figure 11 An example environment of a networked system having a shared augmented reality video call system according to one or more embodiments is shown.

[0020] Figure 12 An example social graph according to one or more implementations is shown. Detailed Implementation

[0021] This disclosure describes one or more embodiments of a shared augmented reality (AR) video calling system that establishes an augmented reality data channel to enable shared AR video calls, which facilitate the sharing and interaction with augmented reality elements among multiple client devices during a video call. Specifically, in one or more embodiments, the shared AR video calling system establishes a video data channel and an AR data channel for video calls between multiple client devices. In practice, in one or more embodiments, the shared AR video calling system enables multiple client devices engaged in a video call to send (or share) AR data between the multiple client devices via the AR data channel during the video call, thereby facilitating the sharing of AR elements and / or interactive AR objects to achieve shared AR environment scenes, AR effects, and / or AR-based games.

[0022] In one or more embodiments, the shared AR video calling system establishes (or utilizes) a multi-peer application programming interface (API) for sharing AR video calls, which enables multiple client devices to send (and receive) AR data in addition to video and audio data during a video call. For example, the shared AR video calling system enables multiple participant client devices (also referred to as participant devices) to send (and receive) AR identifiers via an AR data channel, which provide instructions (or other information) for rendering one or more AR elements within the video captured by the client devices. In some cases, the shared AR video calling system enables multiple participant client devices to send (and receive) AR object vector data, which provides instructions (or other information) for determining the location and / or other spatial information of AR objects as they are rendered within the video captured by the receiving client device. In addition, the shared AR video call system enables multiple participant client devices to send (and receive) AR data, such as, but not limited to, identifiers that specify which video call participant device renders a particular AR element, and / or selectors that specify the main participant client device for a particular AR effect (e.g., the main participant client device for a birthday effect).

[0023] In some embodiments, a shared AR video call system establishes a shared AR video call that enables multiple client devices to send (and receive) AR data during a video call to create or render a shared AR environment scene within the video call. For example, the shared AR video call system can enable multiple participant client devices to send (and receive) AR identifier data, such that all participant client devices render a shared AR environment scene during the video call. For instance, during a video call, when a participant client device initiates an AR-based environment scene, it sends AR data to one or more other client devices in the video call. The one or more other receiving client devices then use the AR data to render the AR-based environment scene within their respective captured video. In this way, multiple participating client devices in the video call render an AR environment scene that is incorporated into the video of each individual participating client device (also referred to as a participant device) in the video call.

[0024] In some cases, shared AR video calling systems also enable multiple participant client devices to interact with the AR environment and / or objects within it, allowing these interactions to be reflected within the AR environment of one or more of the participating client devices. For example, upon interacting with AR elements in a video captured by a first participant client device, the first participant client device can render modified AR elements (due to the interaction) and also send AR data for interacting with the AR elements to a second participant client device in the video call. When the second participant client device receives this AR data via the AR data channel in the video call, it can render modified AR elements that take into account the interactions that occurred on the first participant client device.

[0025] Additionally, in some embodiments, the shared AR video call system enables AR objects to move or transition within the AR environment of a video call between multiple participant client devices. For example, a first participant client device can use an AR object to render a first captured video. The first participant client device can then detect that the AR object in the first captured video is moving to a location outside the viewing frame of the first captured video. The first participant client device can transmit AR data via an AR data channel, which includes information about the AR object, the object vector of the AR object, and a participant identifier for the AR object (e.g., indicating which participant client device will render the AR object). A second participant client device in the video call can then receive the AR data via the AR data channel to render the AR object within a second captured video of the second participant client device using the AR object information, the object vector of the AR object, and the participant identifier of the AR object.

[0026] Furthermore, in some embodiments, the shared AR video call system enables multiple participant client devices to interact with AR activities during a video call. For example, during a video call, multiple participant client devices render and interact with multiple AR objects for one or more AR activities (e.g., AR-based games) by utilizing an AR data channel (established by the shared AR video call system). As an example, a first participant client device may render an AR object (e.g., an ice hockey puck, a ping-pong ball) as part of an AR game within a first video of the first participant client device. Then, in one or more embodiments, the first participant client device may detect user interaction that modifies the AR object (e.g., moving the ice hockey puck, moving the ping-pong ball) and send AR data for the modified AR object to a second participant client device. Subsequently, the second participant client device may detect that the AR object is transitioning to a second video captured by the second participant client device and may render the AR object within the second participant client device (e.g., receiving an ice hockey puck or a ping-pong ball being hit towards the user of the second participant device). In fact, during a video call, the first participant's device and the second participant's device can move AR objects between these devices to interact with AR-based games.

[0027] Furthermore, a shared AR video call system enables multiple participant client devices to send (and render) AR effects to other participant client devices in a video call. For example, a first participant client device can receive a selection of an AR effect and send AR identifier data for that AR effect to a second participant client device via an AR data channel. The second participant device can then receive the AR identifier data for that AR effect and render the AR effect on a second captured video of the second participant device. In particular, in one or more embodiments, the second participant device can render AR effects for interacting with a user depicted in the second captured video (e.g., an AR object depicting a birthday hat being worn by the depicted user, or an AR object depicting an animated character sitting on the shoulders of the depicted user).

[0028] As mentioned above, shared AR video calling systems offer numerous technological advantages and benefits over traditional systems. For example, shared AR video calling systems can establish and implement dynamic and flexible video calls between multiple participant devices, including shared and interactive AR content. In fact, unlike many traditional video calling systems that are limited to presenting AR effects selected by the participant devices for the captured video and streaming captured video depicting non-interactive (overlapping) AR effects to other client devices, shared AR video calling systems enable multiple participant devices to initiate a variety of AR effects (and / or other AR elements) that allow other participant devices in the video call to also render shared AR effects.

[0029] As an example, by utilizing AR data channels between multiple participant devices during a video call, a shared AR video call system enables these multiple participant devices to share AR data (e.g., multiple AR elements, and the corresponding spaces and attributes of these AR elements) among themselves. By enabling the sharing of AR data during a video call, a shared AR video call system facilitates various AR effects and AR elements within the video call. For example, a shared AR video call system allows multiple AR elements to flow freely between renderings of video captured from multiple participant devices, while also responding to interactions detected by these participant devices.

[0030] Beyond the increased flexibility and functionality of AR elements spanning multiple participant devices during video calls, shared AR video call systems also accurately enable the sharing and interaction of AR elements during video calls. For example, by utilizing an AR data channel with a client device layer (which filters or acquires AR data transmitted via the AR data channel during the video call), individual participant devices can render AR data using fully captured video with additional information (e.g., uncompressed video including face tracking information and camera depth information) during the video call, even when the AR effect is initiated by another client device. Furthermore, due to the utilization and filtering of AR data from the AR data channel, participant devices can effectively receive AR data within the video call without being overwhelmed by data from the video call channel, while sending participant devices transmit AR data in real time without needing to identify or determine the receiving participant device.

[0031] As illustrated by the foregoing discussion, this disclosure utilizes various terms to describe the features and advantages of a shared AR video calling system. Additional details regarding the meaning of these terms are now provided. For example, as used herein, the term "video call" refers to electronic communication that transmits video data between multiple computing devices. In particular, in one or more embodiments, a video call includes electronic communication between multiple computing devices that transmit and present video (and audio) captured on these computing devices.

[0032] As used herein, the term "channel" refers to a medium or stream used to transmit data (e.g., data packets) between multiple client devices and / or networks. In some cases, the term "video data channel" can refer to a medium or stream used to transmit video data between multiple client devices and / or networks. In practice, a video data channel can enable the continuous streaming of video data between multiple client devices to display video (e.g., a collection of moving image frames). In some cases, a video data channel may also include audio data for the captured video. Furthermore, the term "audio data channel" can refer to a medium or stream used to transmit audio data between multiple client devices and / or networks, enabling the continuous streaming of audio between multiple client devices to play audio content (e.g., recordings captured from microphones on client devices).

[0033] Additionally, as used herein, the term "augmented reality data channel" refers to a medium or stream used to transmit AR data between multiple client devices and / or networks (for video calls). For example, the term "augmented reality data channel" can enable the continuous streaming (and / or context transfer and / or request) of AR data (e.g., AR elements, AR environment scenes, interactions with AR, AR object vectors) between multiple client devices to deliver AR content and interact with it. In some cases, shared AR video calling systems utilize data exchange formats such as JavaScript Object Notation (JSON), Real-Time Protocol (RTP), and / or Extensible Markup Language (XML) to write, send, receive, and / or read AR data from the AR data channel.

[0034] As used herein, the term "augmented reality element" (sometimes referred to as "augmented reality object") refers to visual content (two-dimensional and / or three-dimensional) displayed (or applied) by a computing device (e.g., a smartphone or head-mounted display) on real-world video (e.g., a live video feed) (e.g., video capturing the real-world environment and / or user during a video call). Specifically, the term "augmented reality element" can include graphical objects, digital images, digital video, text, and / or graphical user interfaces displayed on (or within) a computing device that also renders video or other digital media. For example, an augmented reality element can include graphical objects (e.g., three-dimensional and / or two-dimensional objects) that are interactive, manipulable, and / or configured to realistically interact with the environment (or people) captured in the video from the computing device (e.g., based on user interaction, movement, lighting, shadows). In practice, in one or more embodiments, AR elements can modify the foreground and / or background of a video, and / or modify filters on the video.

[0035] As used herein, the term "augmented reality identifier" refers to information that identifies or addresses a specific AR element. Specifically, the term "augmented reality identifier" can include values ​​(e.g., numbers, alphanumeric characters, addresses, API calls) that identify a specific AR element. For example, upon receiving an AR identifier, a participant's device can use that AR identifier to retrieve and / or identify a specific AR element from the memory and / or library of AR elements (e.g., downloaded to the participant's device during a video call and / or accessible on a cloud service).

[0036] Additionally, as used herein, the term "augmented reality environment scene" refers to one or more augmented reality elements that are interactive, manipulable, and / or configured to interact realistically with each other and / or with user interactions detected on a computing device. In some embodiments, an augmented reality environment scene includes one or more augmented reality elements that modify and / or depict a graphical environment that replaces a real-world environment captured in video on a computing device. As an example, a shared AR video call system can render an augmented reality environment scene to depict one or more participants in a video call as being within a graphical environment (e.g., in space, underwater, at a campfire, in a forest, or on a beach) within a captured video on a computing device. In some cases, a shared AR video call system also enables multiple augmented reality elements within an augmented reality environment scene to be interactive, manipulable, and / or configured to interact realistically with user interactions detected on multiple participant devices.

[0037] Furthermore, as used herein, the term "augmented reality effect" refers to one or more augmented reality elements that present (or display) interactive graphical animation, manipulable graphical animation, and / or spatially aware graphical animation. In particular, the term "augmented reality effect" can include graphical animation that realistically interacts with a person (or user) captured within a video, making the graphical animation appear realistically present in the environment of the person captured in the video. As an example, augmented reality effects can include graphical confetti, graphical hats worn by participants in a video call, and modifications to a person captured within a video call (e.g., wearing a mask, changing the appearance of a participating user in a video call, changing clothing, adding graphical accessories).

[0038] As further used herein, the term "augmented reality object vector" refers to a collection of data representing spatial information of an AR element (e.g., a graphical AR object). Specifically, the term "augmented reality object vector" may include orientation values ​​(e.g., two-dimensional and / or three-dimensional coordinates, angles) and / or magnitude values ​​(e.g., representing velocity and / or acceleration). In practice, in one or more embodiments, a shared AR video call system utilizes AR object vectors to determine the direction of movement of AR elements.

[0039] As used further herein, the term "participant identifier" refers to a value that points to or uniquely identifies a participant device (or participant) within an electronic communication (e.g., a video call). For example, a shared AR video call system utilizes participant identifiers to determine the receiving participant device of a transmitted AR identifier or AR element. In some instances, the participant device includes a participant identifier having an AR element or AR identifier that is used to designate a primary participant device when the AR element or AR identifier is transmitted. The primary participant device can then detect the participant identifier and thus render an AR environment within the captured video using the AR element or AR identifier (e.g., when it is determined that the participant identifier matches an identifier assigned to the primary participant device).

[0040] As further used herein, the term "user interaction" refers to an action or input detected by a participant's device via a camera, touchscreen, and / or computer peripherals (e.g., mouse, keyboard, controller). In some cases, the term "user interaction" includes user input interacting with displayed AR elements. Furthermore, the term "user interaction" can include motion interactions detected by a camera on a client device. For example, motion interactions can include physical movements of the user (e.g., facial movements, arm movements, leg movements) detected by a camera and associated with the location of an AR element. As an example, motion interactions can include, but are not limited to, using a client device's camera to detect a user tapping, hitting, or kicking an AR element. Additionally, motion interactions can include, but are not limited to, using a client device's camera to detect a user's eyes being open, a user blowing on an AR-based object (e.g., blowing out an AR-based candle, blowing away an AR-based leaf), and / or a user biting an AR-based object (e.g., eating AR-based food, using head movements to move an AR-based object).

[0041] Additional details regarding the shared AR video calling system are now provided with reference to the accompanying drawings. For example, Figure 1 A schematic diagram of an exemplary system environment (“Environment”) 100 is shown, in which a shared augmented reality video call system 104 (also referred to as “Shared AR Video Call System 104”) can be implemented. Figure 1 As shown, environment 100 includes one or more server devices 102, network 106, and multiple client devices 108a and 108b-108n.

[0042] although Figure 1 Environment 100 is depicted as having a specific number of components, but environment 100 can have any number of additional components or any number of alternative components (e.g., any number of server devices and / or any number of client devices communicating directly with or via network 106 to the shared AR video call system 104). Similarly, although Figure 1 This illustration shows a specific arrangement of the one or more server devices 102, network 106, and the plurality of client devices 108a and 108b-108n, but various other arrangements are also possible. The one or more server devices 102, network 106, and the plurality of client devices 108a and 108b-108n can be directly communicatively coupled to each other, or indirectly communicatively coupled to each other (e.g., via the following...). Figure 11 and Figure 12(A more detailed discussion of network 106 for communication coupling). Furthermore, the one or more server devices 102 and the plurality of client devices 108a and 108b-108n may include various computing devices (these computing devices include, as discussed above). Figure 10 (One or more computing devices discussed in more detail).

[0043] As mentioned above, environment 100 includes one or more server devices 102. In one or more embodiments, the one or more server devices 102 generate, store, receive, and / or transmit digital data, including digital data relating to video data and AR data for video calls between multiple client devices (e.g., client devices 108a and 108b-108n). In some embodiments, the one or more server devices 102 include a data server. In one or more embodiments, the one or more server devices 102 include a communication server or a web-hosting server.

[0044] like Figure 1 Further shown, the one or more server devices 102 include a shared AR video call system 104. In one or more embodiments, the shared AR video call system 104 establishes video data channels and AR data channels among multiple client devices to enable video calls with shared AR elements among these client devices. In fact, in one or more embodiments, the shared AR video call system 104 establishes video calls with interactive shared AR elements among various numbers of client devices 108a and 108b-108n. Furthermore, in one or more embodiments, the shared AR video call system 104 is implemented as part of a social networking system that facilitates electronic communication such as instant messaging, video calls, and / or social network posts (e.g., regarding...). Figure 11 and Figure 12 (As discussed in more detail).

[0045] Furthermore, in one or more embodiments, environment 100 includes a plurality of client devices 108a and 108b-108n. For example, client devices 108a, 108b-108n may include computing devices capable of interacting with a shared AR video calling system to conduct video calls (and / or other electronic communications) with one or more other client devices. In practice, client devices 108a, 108b-108n may capture video using their digital cameras and may also render AR elements within the captured video using AR data from an AR data channel of the video call. In some embodiments, client devices 108a, 108b-108n include at least one of the following: a smartphone, tablet, desktop computer, laptop computer, head-mounted display device, or other electronic device.

[0046] Additionally, in some embodiments, each of the plurality of client devices 108a and 108b-108n is associated with one or more user accounts in the social networking system (e.g., as per [reference to...]). Figure 11 and Figure 12 (As described). In one or more embodiments, client devices 108a, 108b-108n include one or more applications (e.g., video call applications 110a, 110b-110n) capable of interacting with the shared AR video call system 104 (e.g., interacting with the shared AR video call system 104 by initiating video calls, sending video data and / or AR data, and / or receiving video data and / or AR data). Furthermore, video call applications 110a and 110b-110n are also capable of rendering AR elements within the captured video of a user operating the client device, utilizing video call data and / or AR data. In some instances, video call applications 110a and 110b-110n include software applications installed on client devices 108a and 108b-108n. However, in other cases, video call applications 110a and 110b-110n include web browsers or other applications that access the software applications hosted on the one or more server devices 102.

[0047] The shared AR video call system 104 can be implemented wholly or partially by the various elements in the environment 100. In fact, although... Figure 1The implementation of the shared AR video calling system 104 is shown in relation to the one or more server devices 102, but different components of the shared AR video calling system 104 can be implemented by various devices within environment 100. For example, one or more (or all) components of the shared AR video calling system 104 can be implemented by a different computing device (e.g., one of the plurality of client devices 108a and 108b-108n) or by a server separate from the one or more server devices 102.

[0048] As mentioned above, the shared AR video call system 104 can establish AR data channels between multiple client devices to enable shared AR video calls, which facilitate the sharing and interaction with AR elements during video calls. For example, Figure 2 An example is shown of a shared AR video call system 104 that facilitates the establishment of a shared AR video call among multiple participant devices. For example... Figure 2 As shown, the shared AR video call system 104 establishes a video call stream 202 between client device 204 and client device 205. In fact, as... Figure 2 As further shown, the shared AR video call system 104 establishes a video call stream 202, which includes a video data channel 206, an audio data channel 208, and an AR data channel 210.

[0049] like Figure 2 As shown, client device 204 displays video 214 captured by client device 204 and video 216 captured by client device 205 (and transmitted to client device 204 via video data channel 206) within digital video call interface 212. Similarly, as Figure 2 As shown, client device 205 displays video 216 captured by client device 205 and video 214 captured by client device 204 (and transmitted to client device 205 via video data channel 206) within digital video call interface 218. In some cases, client devices 204 and 205 also transmit audio data captured from their respective client devices via audio data channel 208.

[0050] like Figure 2 As further shown, client devices 204 and 205 transmit (and receive) AR data between themselves during a video call using AR data channel 210 to render shared AR elements. For example... Figure 2As shown, both client device 204 and client device 205 render the shared AR element 220 during the video call. In one or more embodiments, as described below (e.g., refer to...), Figures 3 to 8 Client devices 204 and 205 can interact with (and / or move) the shared AR element 220 that is shared between client devices 204 and 205.

[0051] like Figure 2 As shown, client device 204 renders AR element 220 on the captured video 214. Then, client device 204 sends the captured video 214 with the rendered AR element 220 to client device 205. Similarly, as Figure 2 As shown, client device 205 renders AR element 220 on the captured video 216. Subsequently, client device 205 sends the captured video 216 with the rendered AR element 220 to client device 204. By rendering the AR element locally, the client device can utilize uncompressed captured video, which includes video sensor data, such as, but not limited to, depth information, face tracking information, and / or other body motion tracking information.

[0052] In some embodiments, during a video call, the shared AR video call system 104 utilizes an AR data channel to facilitate the real-time transmission of AR data. For example, during a video call, the shared AR video call system 104 may establish an AR data channel that facilitates the sending (and receiving) of additional data (e.g., in addition to video and audio data) during the video call to share interactive AR environments, AR effects, AR-based activities, and / or individual AR elements. For example, during a video call, the shared AR video call system 104 may enable multiple client devices to send AR data, such as AR identifiers (e.g., AR identifiers pointing to AR elements), AR elements, AR object vectors (for the position and orientation of AR elements), user interactions with AR elements, modifications to AR elements, and / or other updates corresponding to AR environments, AR effects, AR-based activities, and / or individual AR elements.

[0053] In some cases, the shared AR video call system 104 establishes an AR data channel to facilitate the transmission of AR data within the AR data channel using one or more data exchange formats. For example, the shared AR video call system 104 can enable the AR data channel to transmit AR data in various formats, such as, but not limited to, JSON, plain text, and / or XML. Additionally, in one or more embodiments, the shared AR video call system 104 utilizes an end-to-end network protocol to establish the AR data channel, which facilitates the streaming of AR data between multiple client devices in real-time. For example, the shared AR video call system 104 can enable the AR data channel to transmit AR data via an end-to-end network protocol, such as, but not limited to, Real-Time Transport Protocol (RTP), Real-Time Streaming Protocol (RTSP), Real-Time Data Transport (RDT), and / or another data synchronization service.

[0054] Furthermore, in one or more embodiments, the shared AR video call system 104 provides an application programming interface (API) to one or more client devices to communicate with each other and with the shared AR video call system 104 during a video call. Specifically, the shared AR video call system 104 may provide the API, including calls to transmit requests, information, and / or notifications for AR data over an AR data channel (established by the shared AR video call system 104). For example, during a video call, a client device may utilize the API to communicate with the shared AR video call system 104 to request AR effects (or other AR data), thereby initializing the transmitted AR environment, AR effects, AR-based activities, and / or individual AR elements. Additionally, during a video call, a client device may utilize the API to transmit AR data to other client devices via the AR data channel. In fact, according to one or more embodiments herein, during a video call, the shared AR video call system 104 may utilize the API to facilitate communication between the shared AR video call system 104 and one or more client devices to share AR data during the video call, thereby creating AR environments, AR effects, AR-based activities, and / or individual AR elements.

[0055] For example, in one or more embodiments, the shared AR video call system 104 uses JSON-formatted messages broadcast via an AR data channel between multiple client devices to transmit AR data. Specifically, during a video call, the shared AR video call system 104 establishes a data message channel capable of transmitting JSON-formatted messages as an AR data channel. In practice, the shared AR video call system 104 can establish a data message channel that persists throughout one or more AR effects during a video call. Furthermore, the shared AR video call system 104 can establish a data message channel called a bidirectional communication data channel, which facilitates requests to send and receive AR data.

[0056] Furthermore, in one or more embodiments, the shared AR video call system 104 utilizes JSON-formatted messages as JSON objects, which include one or more accessible values. Specifically, the JSON object may include one or more variables and / or one or more data references (e.g., via boolean values, strings, numbers), accessible via calls to specific variables and / or data references. In practice, the shared AR video call system 104 can facilitate the sending and receiving of multiple JSON objects accessed to determine information such as AR identifiers, AR effect lengths, AR object vectors, and / or participant identifiers.

[0057] Furthermore, in some embodiments, the shared AR video call system 104 establishes a data message channel (as the AR data channel) to include a data stream initiated for a client device subscription. For example, the shared AR video call system 104 may receive a request from a client device to subscribe to the AR data channel. The shared AR video call system 104 then enables the client device to listen to data messages transmitted (or broadcast) via the data stream.

[0058] Additionally, the shared AR video call system 104 can establish an AR data channel utilizing a real-time asynchronous data channel (asynchronous channel). In one or more embodiments, the shared AR video call system 104 establishes an asynchronous channel with high throughput and high bandwidth, enabling data to be broadcast in real time to multiple client devices. In some cases, the shared AR video call system 104 establishes an asynchronous channel to broadcast AR data (or AR data messages) regardless of message order. In some instances, the shared AR video call system 104 can establish an AR data channel utilizing a synchronous data channel (synchronous channel). In some embodiments, the shared AR video call system 104 establishes a synchronous channel that transmits AR data (or AR data messages) in message order to synchronize AR data across multiple client devices.

[0059] Furthermore, the shared AR video call system 104 can establish an AR data channel during a video call to transmit AR data having one or more logical data objects (e.g., many-to-one data patches) that provide instructions for performing AR effects. In one or more embodiments, the shared AR video call system 104 enables multiple client devices in a video call to package and / or send logical data objects, receive logical data objects, unpack logical data objects to access logical instructions for performing AR effects, and / or configure the duration and / or timing of packaging, sending, receiving, and / or unpacking events of logical data objects.

[0060] In some instances, the shared AR video call system 104 generates logical data objects as AR data transmitted via an AR data channel. These logical data objects include digital data, Boolean data, text data, or vector data (e.g., vector 2, vector 3, vector 4) of the AR effect instance. In practice, the logical data object may include a key (e.g., for identifying or representing the AR data being transmitted), a participant identifier (e.g., for identifying one or more selected main participants in the video call), and / or an identifier for the AR data channel (e.g., which data channel the logical data object will be sent to).

[0061] In one or more embodiments, the shared AR video call system 104 enables a client device to send multiple logical data objects as AR data, which represent various components of the AR effect. For example, a first logical data object may represent a selected participant in the video call (who is the primary participant in the AR effect). Additionally, a second logical data object may represent location data for the AR effect (e.g., a location-based object vector or a specific object selection). As an example, location data may include a direction-based object vector or a specific object selection, such as, but not limited to, a face tracker selection, an eye tracker selection, and / or a nose tracker selection. In practice, the shared AR video call system 104 enables a client device to send various combinations of multiple logical data objects during a video call to render various combinations of shared AR effects on another client device.

[0062] Furthermore, the logical data object may include one or more AR element identifiers for specifying one or more AR elements to be rendered. Additionally, the logical data object may include modifiers, timers, and / or actions that specify the duration of the AR element (e.g., a throttle object) and / or specific rendering instructions for the AR element (e.g., position, velocity, quantity). Furthermore, the logical data object may also include one or more links that temporally and / or logically link various information and / or instructions within the logical data object, allowing the client device to utilize temporal and / or logical links to determine how to render the AR object. In fact, as described in more detail below (e.g., regarding...), Figures 6A to 6D During a video call, the shared AR video call system 104 can utilize multiple logical data objects as AR data to achieve one or more of multiple shared AR effects.

[0063] To illustrate an example of how the shared AR video call system 104 utilizes logical data objects to achieve shared AR effects during a video call, the shared AR video call system 104 enables a first client device to detect a kissing gesture made by a participant on the first client device during the video call. This allows a second client device to receive an AR data object (which includes logical data objects) that causes the second client device to render an AR heart (e.g., an AR element) on the nose of the participant captured on the second client device. In practice, as part of the AR effect, the shared AR video call system 104 can detect a kissing gesture being made by a participant captured on the first client device (e.g., using a face tracker component on the video call).

[0064] Upon detecting a kissing action, the first client device can detect the number of kisses and generate a logical data object. This logical data object includes an identifier for the kissing event (e.g., an AR identifier indicating the rendering of an AR effect for that kissing event) and a counter for the number of kisses. The first client device can then send this logical data object to a second client device via an AR data channel. Furthermore, the second client device can unpack the logical data object to identify the identifier for the kissing event and the counter.

[0065] As part of the logical data object, the second client device can also identify instructions for the duration corresponding to the AR effect, AR element identifiers, and instructions for rendering AR elements (e.g., throttling objects). In this regard, the second client device can render AR elements identified by the logical data object (e.g., AR hearts) at a specified location (e.g., a static location within the video captured by the second client device via instructions for tracking the nose or other features of a participant on the second client device) for a specified duration and a specified number of times (based on a counter) to a specified location.

[0066] In some cases, such as Figure 2 As shown, the client device includes a client device layer for the video call stream (established by the shared AR video call system 104). Specifically, the client device may utilize a client device layer (e.g., a layer within an API and / or a network protocol) to control the transmission and / or reception of AR data via an AR data channel. For example, the client device may utilize the client device layer to receive and / or filter AR data broadcast (or transmitted) via an AR data channel from one or more client devices participating in the video call. Specifically, in one or more embodiments, multiple client devices transmit AR data to each client device participating in the (same) video call via an AR data channel. Furthermore, the client device may utilize the client device layer to identify the transmitted AR data and filter it (e.g., to utilize or ignore AR data). For example, the client device may utilize the client device layer to filter AR data based on a participant identifier corresponding to the AR data (as described below).

[0067] like Figure 2 As shown, in some cases, the shared AR video call system 104 can enable shared AR video calls among multiple client devices. For example, as Figure 2 As shown, the shared AR video call system 104 can establish a video call stream 202 (e.g., video data channel 206, audio data channel 208, and AR data channel 210) among client devices 204, 205, and one or more of a plurality of client devices 222. In practice, once AR data is transmitted via AR data channel 210, client devices 204, 205, and one or more of the plurality of client devices 222 can render shared AR elements to create shared AR environment scenes, AR activities, and / or AR effects (according to one or more embodiments herein) within the video call among the multiple client devices.

[0068] In fact, as mentioned above, in one or more embodiments, the shared AR video call system 104 establishes an AR data channel during a video call for transmitting (or conveying) AR data to create interactive and free-flowing AR elements between multiple client devices within the video call. Figure 3 A flowchart illustrates how an AR video call system 104 establishes an AR data channel during a video call, and how multiple client devices utilize this AR data channel to transmit AR data (thereby rendering AR elements). For example, as... Figure 3 As shown, in action 302, the shared AR video call system 104 receives a request from client device 1 to conduct a video call with client device 2 (e.g., a request to initiate a video call). Then, as... Figure 3 As shown in action 304, the shared AR video call system 104 establishes a shared AR video call (e.g., including a video data channel, an audio data channel, and an AR data channel) between client device 1 and client device 2.

[0069] Then, as Figure 3 As shown in action 306, client device 1 sends a first video stream (e.g., a video stream captured on client device 1) to client device 2 via a video data channel and an audio data channel. Figure 3 As further shown in action 308, client device 2 sends a second video stream (e.g., a video stream captured on client device 2) to client device 1 via a video data channel and an audio data channel. Additionally, as... Figure 3 As shown in action 310, client device 1 renders the first video stream and the second video stream. Similarly, as... Figure 3 As shown in action 312, client device 2 also renders the first video stream and the second video stream.

[0070] like Figure 3 As further shown in action 314, client device 1 initiates the sharing of AR elements within the first video stream of client device 1. Additionally, as... Figure 3 As shown in action 316, client device 1 sends (or transmits) the AR identifier of the AR element to client device 2 (via the AR data channel). Furthermore, as... Figure 3 As shown in action 318, client device 2 renders the first video stream using AR elements. For example... Figure 3 As shown in action 320, in response to receiving an AR identifier, client device 2 renders a second video stream using AR elements associated with the received AR identifier. In fact, as described below (e.g., regarding...) Figures 4 to 8 The video streams of client device 1 and client device 2 each render shared AR elements locally to create shared AR environment scenes, AR activities, and / or AR effects.

[0071] In some embodiments, when these client devices share AR elements on an AR data channel, these client devices initialize the AR elements before rendering them. Specifically, such as... Figure 3 As shown in actions 322a and 322b, multiple client devices initialize the AR element before rendering it. In one or more embodiments, one or more client devices receive an AR identifier and wait until each client device initializes the AR element to synchronize the shared AR experience across multiple client devices in a video call. In practice, once initialized, these client devices can send (via the AR data channel) a message to other client devices indicating that the client device is ready to render the AR element (or has initialized the AR element). In some cases, client devices can initialize the AR element by retrieving the AR element, loading the AR element, and / or downloading a specific AR data package for rendering the AR element. Once an initialization message is received from each client device in the video call, each client device can proceed to render the AR element.

[0072] Furthermore, in one or more embodiments, multiple client devices (or the shared AR video call system 104) pause one or more video streams of the video call until the shared AR element is initialized on each of the multiple participating client devices. In some cases, the shared AR video call system 104 streams (or displays) a loading screen (or loading animation) until the shared AR element is initialized on each of the multiple participating client devices. Additionally, in one or more embodiments, while the shared AR element is initialized on multiple client devices, the shared AR video call system 104 enables these client devices to continue sending and / or receiving audio data via an audio data channel and to play audio from one or more client devices during the video call.

[0073] like Figure 3 Action 324 further illustrates that client device 1 interacts with AR elements within the first video stream (i.e., the video stream captured by client device 1). Upon interacting with the AR elements, client device 1 also sends an additional AR identifier (such as...) to client device 2 for this interaction. Figure 3 (As shown in action 326). Then, as... Figure 3 As shown in action 328, client device 2 receives the additional AR identifier and uses the information from the additional AR identifier used for interaction (from client device 1) to modify (or add) the AR element in the second video stream based on the additional AR identifier (representing the interaction from action 324). For example, as described below (e.g., regarding...) Figure 4Client device 2 can modify AR elements. In some cases, as described below (e.g., regarding...), Figure 5 , Figure 6A and Figure 6B Client device 2 uses information from the attached AR identifier to add AR elements to (or remove AR elements from) the second video stream.

[0074] In one or more embodiments, multiple client devices (reference) Figure 3 Various combinations of data (e.g., via video data channels, audio data channels, and / or AR data channels) can continue to be transmitted between these client devices during a video call. In fact, these multiple client devices can transmit various combinations of the following data during a video call: video data, audio data, and / or AR data to present various AR experiences. Furthermore, although... Figure 3 A flowchart illustrating data transmission between two client devices interacting within a video call is shown. However, the shared AR video call system 104 can establish various combinations of video data, audio data, and / or AR data to transmit between a variety of client devices during a video call to present various AR experiences. Furthermore, one or more of the multiple client devices (client devices participating in the video call) can change the AR environment, AR activity, and / or AR effect during the video call by selecting a different AR environment, a different AR activity, and / or a different AR effect.

[0075] As mentioned above, during a video call, the shared AR video call system 104 utilizes an AR data channel for transmitting AR data between multiple client devices participating in the video call to establish a shared AR environment scene. For example, Figure 4 This illustrates a shared AR video call system 104 that establishes video calls with a shared AR environment scene between multiple client devices. For example... Figure 4 As shown, once a request to initiate a video call is received from client device 402a (or client device 408a), the shared AR video call system 104 establishes a video call stream 420 including a video data channel 422, an audio data channel 424, and an AR data channel 426.

[0076] like Figure 4 As further shown, client device 402a sends an AR identifier via AR data channel 426 and renders AR element 414a within the captured video 406 in digital video call interface 404. Upon receiving the AR identifier, client device 408a (e.g., ...) Figure 4 (As shown) AR element 414b is also rendered within the captured video 412 in the digital video call interface 410. Similarly, as... Figure 4 As shown, client devices 402a and 408a also display video streams depicting the following (e.g., during a video call): captured video 412 with AR element 414b, and captured video 406 with AR element 414a. In fact, regarding... Figure 4 Shared AR elements create an AR environment scene with a single cake (e.g., via AR elements 414a and AR elements 414b) across multiple client devices during a video call.

[0077] While one or more embodiments illustrate a shared AR video call system 104 establishing a video call (in which a client device initiates a shared AR environment scene displaying an AR-based cake), the shared AR video call system 104 enables these client devices to send a variety of AR identifiers (for multiple AR elements) to create various shared AR environment scenes within the video call. As an example, in some cases, the client device sends one or more AR identifiers referring to AR elements within an AR campfire environment within the video call. In practice, multiple client devices in the video call can receive these one or more AR identifiers to render the AR elements, thereby creating a shared graphical AR campfire within the video call.

[0078] As another example, in some cases, the client device sends one or more AR identifiers that refer to AR elements representing an AR outer space environment within the video call. Therefore, multiple client devices in the video call can receive these one or more AR identifiers to render the AR elements, thereby creating a shared graphical AR outer space environment within the video call (e.g., to place participants in the video call in the same outer space environment). Furthermore, in some embodiments, the client device sends one or more AR identifiers that refer to AR elements representing an AR underwater environment within the video call. In practice, multiple client devices in the video call can receive these one or more AR identifiers to render the AR elements, thereby creating a shared graphical underwater environment within the video call (e.g., to place participants in the video call in the same underwater environment).

[0079] As mentioned above, the shared AR video call system 104 enables multiple client devices (or users of multiple client devices) to interact with AR elements within the video call, thereby influencing AR elements rendered on one or more of the participating client devices during the video call. For example, Figure 4 It also demonstrates how a client device detects interaction with an AR element and how this interaction is reflected on other client devices during the video call using an AR data channel. In fact, as... Figure 4 As shown, client device 402b (an interim version of client device 402a) detects user interaction 418 within the captured video 406 (e.g., the captured participant blowing out the candle of AR element 414a representing a cake) to render the modified AR element 416a.

[0080] Then, as Figure 4 As further shown, client device 402b sends an AR identifier with interactive data (used to indicate the modified AR element and / or interaction with the AR element) to client device 408b (an interim version of client device 408a) via AR data channel 426. Upon receiving the AR identifier indicating the modified AR element and / or interaction with the AR element, client device 408b also renders AR element 414b within the captured video 412 to reflect the interaction detected on client device 402b as the modified AR element 416b.

[0081] Similarly, Figure 4 As shown, client devices 402b and 408b display video streams used to depict captured video 412 with modified AR element 416b on client device 402b and captured video 406 with modified AR element 416a on client device 408b (e.g., during a video call). In fact, as... Figure 4 As shown, since user interaction 418 (e.g., mobile interaction) is detected on the video 406 captured on client device 402b, both client device 402b and client device 408b display (respectively) the modified AR elements 416b and 416a (as a cake with extinguished candles). Figure 4 As shown, the shared AR video call system 104 establishes a video call in which participating users of multiple client devices can interact with AR elements, such that these interactions create and manipulate a shared AR environment during the video call to generate the perception that the video call is taking place in the same physical location (e.g., in the same room or space).

[0082] In one or more embodiments, the shared AR video call system 104 enables multiple client devices to detect and transmit various types of interaction data regarding AR elements. For example, the interaction data may include multiple interaction types, such as, but not limited to, hand movements, arm movements, head movements, eye movements, body movements, and touchscreen inputs associated with (and / or interacting with) one or more AR elements. Additionally, the interaction data may include location data of the AR element with which the interaction occurs (e.g., the portion of the AR element with which the interaction occurs). In practice, in some embodiments, the interaction data is applied to the AR element to determine the reaction state of the AR element based on that specific interaction data (e.g., a specific action causes a specific reaction state of the AR element). In practice, client devices may receive (via an AR data channel) the interaction data and use it to determine the reaction state of the AR element caused by the interaction data, and render a modified AR element accordingly.

[0083] In some instances, the client device can use locally detected interaction data to determine the reaction state of an AR element and render a modified AR element accordingly. The client device can then send (via an AR data channel) the modified state of the AR element to other client devices in the video call, indicating the modification, location, and / or state of the AR element. Furthermore, these other client devices can receive information about the modified state of the AR element from the AR data channel and render a modified AR element with a similar state to that identified by the AR data received via the AR data channel.

[0084] In one or more embodiments, the shared AR video call system 104 enables multiple client devices to send various types of AR data (AR identifiers for multiple AR elements) that indicate various interactive data and / or modifications to AR elements to create various shared AR interactive environment scenes within a video call. As an additional example, in some instances, a client device detects hand movement interacting with one or more AR-based balloons (e.g., as AR elements) rendered on a video captured by the client device. In response to the detected hand movement, the client device renders one or more of these AR-based balloons as exploding (e.g., a modified AR element) on the video captured by the client device. Additionally, the client device sends this interaction (and / or the modified state of the AR-based balloons) to another client device in the video call via an AR data channel. This other client device can receive the AR data and also render the same one or more AR-based balloons as exploding within the shared AR environment.

[0085] As another example, a client device can detect an action that instructs a video call participant to turn a page of an AR-based book rendered on the video captured by the client device. The client device can then render the AR-based book page turned while simultaneously sending this interaction (and / or the modified state of the AR-based book) to another client device in the video call via an AR data channel. The other client device can receive the AR data and also render the same AR-based book page turned within the shared AR environment.

[0086] In some cases, during a video call, the shared AR video call system 104 can, in response to detecting interaction data on another client device, enable AR elements to be rendered (or presented) on one client device. For example, a client device can detect interaction data from participants in the captured video (e.g., waving, clapping, blowing a kiss). The client device can then send the detected interaction data (and additional AR data) to one or more other client devices participating in the video call via an AR data channel. Upon receiving the interaction data (and the additional AR data), the one or more other client devices can render AR elements based on the interaction data and the additional AR data. For example, as mentioned above, upon receiving interaction data for a kissing action, the client device can render an AR element (e.g., an AR heart) at a specified location within the video captured by the client device based on the AR data. As another example, upon receiving interaction data instructing clapping on a first client device (and AR data for the resulting AR effect), a second client device can render AR-based fireworks within the video captured by the second client device.

[0087] As further mentioned above, the shared AR video call system 104 can utilize the AR data channel of the video call to enable AR elements to move between multiple client devices in the video call. Specifically, in one or more embodiments, the shared AR video call system 104 enables AR elements rendered in a captured video on one client device to move to another captured video (e.g., rendered by that other captured video) on another client device in the video call. For example, Figure 5 An AR video call system 104 is shown that allows AR elements to be moved from one client device to another during a video call.

[0088] like Figure 5 As shown, the shared AR video call system 104 establishes a video call between client device 502a and client device 508a by establishing a video call stream 518 including a video data channel 520, an audio data channel 524, and an AR data channel 526. Figure 5 As shown, client device 508a renders AR element 516a within the video 512 captured on client device 508a on digital video call interface 510. In fact, as... Figure 5 As shown, client device 502a receives video data to display the captured video 512 (from client device 508a) with AR element 516a on digital video call interface 504.

[0089] In addition, such as Figure 5 As shown, the client device 502a also captures video 506 and displays the captured video 506 within the digital video call interface 504. Furthermore, as... Figure 5 As shown, client device 508a receives video data to display the captured video 506 from client device 502a on the digital video call interface 510. Similarly... Figure 5 As shown, client devices 502a and 508a also render additional AR elements (as described above) for the AR campfire environment scene during video calls.

[0090] In addition, refer to Figure 5 The client device 508a can track the movement of the AR element 516a within the captured video 512. Then, as... Figure 5 As shown, once it is detected that AR element 516a is exiting (or leaving) the frame of the captured video 512, the client device 508a can send AR data indicating the movement of the AR element (using object vector indication). For example, as Figure 5 As shown, when AR element 516a leaves the screen of client device 508a, client device 508b (the transition of client device 508a) sends AR data via AR data channel 526. This AR data includes the object vector and AR identifier of AR element 516a. Then, as... Figure 5 As further shown, client device 502b (a transition of client device 502a) receives the AR data and renders AR element 516b within the captured video 506 using an AR identifier and an object vector (for correctly locating AR element 516b) to show that AR element 516b is entering the frame of the captured video 506.

[0091] Similarly, Figure 5 As shown, client device 502b captures video 506 with AR element 516b and displays the captured video 506 within digital video call interface 504. Furthermore, as... Figure 5As shown, client device 508b receives video data to display the captured video 506 (from client device 502b) with AR element 516b on digital video call interface 510. In fact, as... Figure 5 As shown, the shared AR video call system 104 enables multiple client devices to transmit AR data between these client devices during a video call, in order to move AR elements between these client devices (e.g., as shown in the image). Figure 5 (The image shows an AR bird flying around in different captured videos during a video call).

[0092] In one or more embodiments, the shared AR video call system 104 enables client devices to broadcast AR elements to other client devices in a video call using AR data via an AR data channel. Specifically, in some instances, the client device determines that the AR element is exiting the frame of the captured video (e.g., by programmable or AI-based movement of the AR element and / or by movement caused by user interaction with the AR element). Once it is determined that the AR element is exiting the frame of the captured video, the client device broadcasts AR data via the AR data channel, which notifies other client devices in the video call that the AR element is entering the frame (or multiple frames) of the captured video (or multiple frames) of one or more other client devices. In practice, in one or more embodiments, the one or more other client devices receive the AR data via the AR data channel and render the AR element (e.g., render it to enter or join the captured video (or multiple frames) of the one or more other client devices).

[0093] In some instances (as mentioned above), the client device may also send object vectors using AR data for the broadcast AR elements. In practice, in one or more embodiments, the object vector includes data indicating the location of the AR element (e.g., the position where the AR element is entering the frame of the captured video, the position where the AR element will soon be present in the captured video). Additionally, the object vector may include motion data, such as, but not limited to, the speed of movement of the AR element (e.g., travel speed, acceleration), and / or other trajectories of movement of the AR element or physics-based data (e.g., the coefficient of friction, centripetal acceleration, gravity factor of the AR element). In practice, the receiving client device can use the object vectors of the AR elements to render the movement of the AR elements to depict a smooth transition from the video previously captured by the client device to the video captured by this client device.

[0094] In some embodiments, during the broadcast of an AR element, the shared AR video call system 104 also enables multiple client devices in a video call to send participant identifiers using AR data, which are used to indicate one or more receiving client devices. For example, when more than two client devices are conducting a video call, one client device sends (or utilizes) a participant identifier as part of the transmitted (or broadcast) AR element to specify one or more target client devices for that AR element. For illustration, a first client device (during a video call with a second and third client device) may send AR data in a manner that includes addressing the participant identifier of the second client device to broadcast (or move) the AR element from the video captured by the first client device to the video captured by the second client device (but not the third client device). In practice, the second and third client devices may filter the AR data from the AR data channel to utilize AR data corresponding to the correct participant identifier. In some cases, AR data transmission may include multiple participant identifiers, such that multiple client devices based on these participant identifiers render the AR element of the AR data transmission. In some cases, the shared AR video calling system 104 can assign a participant identifier (e.g., numeric and / or alphanumeric values) to each participating client device while establishing a video call between client devices.

[0095] In some cases, the shared AR video call system 104 enables AR elements to be randomly moved between multiple client devices during a video call. For example, the shared AR video call system 104 (as a controller) may include the selection of random identifiers for AR data used for broadcast AR elements to set a random destination for the broadcast AR elements. For example, the shared AR video call system 104 may identify AR data within an AR data channel that is broadcasting AR elements from a first client device (which participates in a video call between the first, second, and third client devices). Then, in one or more embodiments, the shared AR video call system 104 may assign the broadcast AR element to a third client device such that the third client device renders the AR element, while the second client device does not render the AR element (e.g., based on filtering the AR data channel). In some instances, during a video call, the shared AR video call system 104 utilizes an array having a randomized list of identifiers from multiple client devices participating in the video call, and selects, via this array, the broadcast AR element randomly originating from one client device to be randomly assigned to another client device.

[0096] In one or more embodiments, the shared AR video call system 104 enables multiple client devices to send various types of AR data (in terms of AR identifiers of AR elements) indicating the movement of various AR elements between the multiple client devices during a video call. As an example, client devices can send AR data via an AR data channel (established by the shared AR video call system 104) indicating the movement of multiple AR-based fish in an AR underwater environment shared among the multiple client devices during the video call. In practice, one or more client devices in the video call can receive AR data indicating the movement of the multiple AR-based fish and render these AR-based fish at specific locations (and / or render images of these AR-based fish entering captured video on these client devices). In effect, the shared AR video call system 104 can enable the movement of various AR elements between multiple client devices during a video call via an AR data channel.

[0097] As another example, a client device can detect interaction with a rendered AR element that moves the AR element within the client device's captured video (e.g., a movement interaction of an AR-based cake moving from left to right on the captured video). In response, the client device broadcasts the movement information of the AR element to other client devices in the video call via an AR data channel. Upon receiving the movement information of the AR element, these other client devices can also render the AR-based cake at a new location to match the movement detected on the client device. In fact, the shared AR video call system 104 can implement the movement of various AR elements, allowing the movement to be mimicked in the AR rendering of other client devices in the video call, thereby generating a perception of the shared AR environment.

[0098] As mentioned above, during a video call, the shared AR video call system 104 can use AR data channels to enable AR-based activities between multiple client devices. For example, Figure 6A and Figure 6B A shared AR video call system 104 is illustrated, which uses an AR data channel to enable AR-based activities between multiple participating users (via corresponding client devices) in a video call. For example... Figure 6A As shown, the shared AR video call system 104 establishes a video call between client devices 602a, 602b, and 602c by establishing a video call stream 612 that includes a video data channel 614, an audio data channel 616, and an AR data channel 618. Figure 6AAs further shown, client device 602a captures video 604a and displays video 604a in digital video call interface 606; client device 602b captures video 604b and displays the captured video 604b in digital video call interface 608; and client device 602c captures video 604c and displays video 604c in digital video call interface 610. Additionally, as... Figure 6A As shown, each of the client devices 602a, 602b and 602c receives video data from these participating client devices via video data channel 614 (and audio data from these participating client devices via audio data channel 616) to display the client device video stream (e.g. to establish a video call).

[0099] In addition, such as Figure 6A As shown, client device 602a renders AR element 620a (e.g., an AR-based ring) within the captured video 604a (which is also streamed on client devices 602b and 602c). Figure 6A As further shown, client device 602a detects a motion interaction related to AR element 620a, which originates from a user being filmed within the captured video 604a. In some embodiments, client device 602a renders AR element 620a such that AR element 620a moves in response to the motion interaction from the user being filmed within the captured video 604a.

[0100] As from Figures 6A to 6B As shown in the transition, AR element 620a is moved to the frame of captured video 604a in the digital video call interface 606 of client device 602a. In response, client device 602a broadcasts AR element 620a via AR data channel 618 using AR data (which indicates the AR identifier of AR element 620a and the object vector of AR element 620a) to move AR element 620a to another client device in the video call (as described above). Specifically, as Figure 6B As shown, client device 602b receives AR data via AR data channel 618, which indicates AR element 620a (from...). Figure 6A The client device 602b uses the AR data to render the AR element 620b entering the captured video 604b in the digital video call interface 608 on the client device 602b, along with the AR identifier and the object vector for the position and orientation of the AR element 620a.

[0101] like Figure 6BAs further shown, the client device 602b renders the AR element 620b at the location indicated by the object vector from the AR data channel 618. Then, as... Figure 6B As shown, client device 602b detects motion interactions between a participant user being filmed and AR element 620b within the captured video 604b. In practice, in one or more embodiments, client device 602b renders AR element 620b, which moves according to the detected motion interactions between the participant user and AR element 620b on the captured video 604b. Furthermore, once AR element 620b is detected exiting the frame of captured video 604b, client device 602b can broadcast AR element 620b to another client device in the video call (e.g., client devices 602c and / or 602a). Figure 6A and Figure 6B As shown, during a video call, the shared AR video call system 104 enables interaction with AR elements (and movement of AR elements) between multiple client devices making the video call, in order to create AR-based activities between multiple client devices.

[0102] In some instances, refer to Figure 6B Both client devices 602b and 602c receive AR element 620a (from AR data channel 618) via AR data channel 618. Figure 6A AR data. As explained above, the shared AR video call system 104 enables multiple client devices to send participant identifiers, which are used to specify one or more target client devices for one or more AR elements. Figure 6B In the example, client device 602a sends a participant identifier to the receiving client device that designates client device 602b as the AR identifier, and therefore, client device 602b renders the AR element 620b within the captured video 604b. Similarly, see reference... Figure 6B The client device 602c detects AR data within the AR data channel and filters out AR data that does not include a participant identifier designating the client device 602c as the receiving client device. In this respect, as... Figure 6B As shown, client device 602c does not take any action to render new AR elements from AR data sent by client device 602a via AR data channel 618.

[0103] In one or more embodiments and referenced Figure 6BClient devices 602a, 602b, and 602c utilize AR data, including one or more logical data objects (as described above), during video calls to enable AR elements (e.g., AR-based rings) to move from one client device to another. As an example, Figure 6C Multiple logical data objects are shown that are used by a client device (e.g., client device 602a) during a video call and transmitted as AR data via AR data channel 618 to achieve a shared AR effect. Furthermore, Figure 6D Multiple logical data objects are shown that are received and / or utilized by other client devices (e.g., client devices 602b and 602c) during a video call to receive and render shared AR effects.

[0104] For example, such as Figure 6C As shown (and referenced) Figure 6B The client device 602a utilizes multiple logical data objects to track faces in the captured video 604a (e.g., via a Face Finder logical data object, a Face Select logical data object, and a Face Tracker0 logical data object). Once the face tracking logical data object detects that the participant's nose is leaving the frame of the captured video 604a, the client device 602a activates a Nose Movement Off-Screen logical data object. This Nose Movement Off-Screen logical data object initiates the packaging of AR data (e.g., pulse packets) and also determines the nose's position object vector (e.g., a nose position vector NosePositionV with x, y, and / or z coordinates). In practice, as... Figure 6C As shown, client device 602a packages logical data objects for nose position (e.g., a pulse pack for "NosePosition"). Then, as... Figure 6C As shown, client device 602a transmits data to AR data channel 618 (e.g., via logical data object) through a logical data object. Figure 6B As shown, AR data is sent, and the logical data object includes an AR identifier (e.g., subject: SendDonut), packaged location data, and a participant identifier (“Participant ID”).

[0105] Subsequently, as Figure 6D As shown (and referenced) Figure 6B Client devices 602b and 602c receive (e.g., multipeer receive) data from AR data channel 618 (e.g., ...). Figure 6B The AR data (shown) includes a logical data object indicating an AR identifier (e.g., subject: SendDonut) and a participant identifier (e.g., “Participant ID”). In one or more embodiments, the receiving client device may utilize the participant identifier to determine whether to unpack the AR data. For example, as in Figure 6B As described, client device 602b matches the participant identifier, while client device 602c does not. In this respect, client device 602b unpacks the AR data shown in Figure 6 to render it. Figure 6B The AR effect shown.

[0106] For example, upon unpacking the AR data packet (e.g., pulse unpacking), client device 602b identifies the object position vector (e.g., NosePositionV) of the AR identifier "SendDonut". Using this AR identifier, client device 602b also utilizes a switch logic data object that causes the AR element ("Donut") corresponding to the AR identifier "SendDonut" to be rendered (or the visibility of the AR element to be switched) at the position indicated by the logic data object of the object position vector (e.g., NosePositionV) within the captured video 604b. In fact, as... Figure 6B As shown, client device 602b utilizes one or more received logical data objects ( Figure 6D (As described in the text) an AR-based ring is rendered into the screen of the client device 602b within the captured video 604b.

[0107] In one or more embodiments, the shared AR video call system 104 enables one or more client devices in a video call to interact with, move, and / or manipulate AR elements on the client devices (as described above) using object vectors and / or participant identifiers to establish AR activities. Furthermore, in some cases, the shared AR video call system 104 also enables other AR data to facilitate AR-based activities between multiple client devices in the video call. For example, the shared AR video call system 104 enables other AR data, such as, but not limited to, score information, AR element size information, and / or other game-based mechanisms (e.g., lives, number of items, timers, speed). In some embodiments, the client devices transmit such AR data via an AR data channel to cause other client devices in the video call to render AR elements to display (or depict) information corresponding to AR-based activities (e.g., AR games).

[0108] Furthermore, the shared AR video call system 104 enables client devices to send various types of AR data (e.g., AR identifiers of AR elements, object vectors, and / or participant identifiers) during video calls to create a variety of interactive and mobile AR elements to facilitate AR-based activities (e.g., AR games). For example, one or more client devices can send and / or receive (via an AR data channel) AR elements during a video call to facilitate interaction with a graphical puck in an AR-based air hockey game (e.g., where, during the video call, the AR-based puck moves on multiple client devices based on motion interactions detected on those client devices).

[0109] Similarly, in one or more embodiments, one or more client devices may send and / or receive (via an AR data channel) AR elements during a video call to facilitate interaction with a graphical ping-pong ball in an AR-based ping-pong game (e.g., where, during the video call, the AR-based ping-pong ball moves on multiple client devices based on motion interactions detected on those client devices). In some cases, one or more client devices may send and / or receive (via an AR data channel) AR elements during a video call to facilitate interaction with AR-based text answer options for a presented question (e.g., a math problem, a trivia question), such that motion interactions detected on the client devices can indicate the user's selection of an AR-based text answer option.

[0110] While one or more embodiments herein illustrate a shared AR video call system 104 enabling AR-based games (as AR activities) between multiple client devices in a video call using an AR data channel, the shared AR video call system 104 can enable various types of AR activities between multiple client devices in a video call. For example, in one or more embodiments, the shared AR video call system 104 enables the transmission of AR data between multiple client devices in a video call (as described above) to create AR activities, such as, but not limited to, shared AR drawing activities, shared AR sketching activities, shared AR whiteboards, and / or shared AR puzzle activities.

[0111] For example, during a video call, one or more client devices can use AR data via an AR data channel to render a shared AR painting environment. User interactions from multiple client devices in the video call involve interacting with and modifying an AR-based canvas containing painting materials and other art tools to create a shared AR painting within the video call. As another example, during a video call, one or more client devices can use AR data via an AR data channel to render a shared AR whiteboard. User interactions from multiple client devices in the video call involve interacting with and modifying the AR-based whiteboard to share notes and / or opinions within the shared AR whiteboard within the video call.

[0112] Furthermore, as mentioned above, the shared AR video call system 104 can achieve AR effects across multiple client devices during a video call. For example, Figure 7 This illustrates how a shared AR video call system 104 enables shared AR effects across multiple client devices during a video call. Specifically, such as... Figure 7 As shown, a client device can send AR data (e.g., an AR identifier) ​​to other client devices in a video call via an AR data channel to initiate AR effects during the video call. In practice, once the AR data is received during the video call, one or more client devices will display the AR effect.

[0113] For example, such as Figure 7 As shown, the shared AR video call system 104 establishes a video call between client devices 702a, 702b, and 702c by establishing a video call stream 712 that includes a video data channel 714, an audio data channel 716, and an AR data channel 718. Figure 7 As further shown, client device 702a captures video 704a and displays video 704a in digital video call interface 706, client device 702b captures video 704b and displays video 704b in digital video call interface 708, and client device 702c captures video 704c and displays video 704c in digital video call interface 710. Additionally, as... Figure 7 As shown, each of the client devices 702a, 702b and 702c receives video data from the participant client devices via video data channel 714 (and audio data from the participant client devices via audio data channel 716) to display the client device video stream (e.g. to establish a video call).

[0114] In addition, such as Figure 7As shown, client device 702a transmits AR data via AR data channel 718, the AR data including AR identifiers for AR effects. Then, as... Figure 7 As shown, client device 702a renders AR element 715a for the AR effect within the captured video 704a (which is also streamed on client devices 702b and 702c). Furthermore, refer to... Figure 7 Client devices 702b and 702c also receive the AR identifier used for the AR effect via AR data channel 718. In response, as... Figure 7 As shown, client device 702b renders AR elements 720 for AR effects within the captured video 704b (which is also streamed on client devices 702a and 702c). Furthermore, as... Figure 7 As shown, client device 702c renders AR elements 715b for AR effects within the captured video 704c (which is streamed on client devices 702a and 702b).

[0115] like Figure 7 As shown, client devices 702a, 702b, and 702c render AR elements within the captured video of a video call to render an AR-based birthday scene. In some cases (such as...) Figure 7 As shown), the AR effect can include rendered AR elements that differ across client devices (e.g., when using the same AR effect theme). For example, as... Figure 7 As shown, the AR effect enables client device 702b to render an AR element 720 depicting a birthday message on the captured video 704b, while other client devices (e.g., client devices 702a and 702c) render related AR elements (e.g., AR-based celebratory confetti) for the AR-based birthday message in their respective captured videos.

[0116] In some embodiments, the client device sends AR data including AR effects. For example, AR effects may include a set of AR-based instructions, AR-based animations, and / or AR-based elements. Upon receiving an AR effect and / or an AR identifier referring to the AR effect, the client device can render AR elements from the AR effect using AR-based instructions and / or animations of the AR elements. For example, AR-based instructions and / or animations may include instructions for positioning the AR element at a specific location (e.g., at the top of a video, on the face of a participant user within the video, or as a wearable item on a participant user within the video). Additionally, AR-based instructions and / or animations may also include instructions for specific movement of the AR element (e.g., speed, specific actions such as crawling, falling, and / or splashing). In practice, the shared AR video call system 104 enables multiple client devices to send AR effects with various combinations of AR elements, AR instructions, and / or AR animations. Furthermore, in one or more embodiments, as described above (e.g., regarding...), Figures 3 to 6D AR elements with AR effects can also be interactive and / or manipulable.

[0117] In some instances, the client device also includes a participant identifier that is sent along with the AR data for AR effects during the video call. For example, as described above, the client device may send a participant identifier to designate one or more target client devices for the AR effect. Additionally, in some embodiments, the client device may send a participant identifier to designate (or differentiate) multiple AR effects among multiple client devices in a video call. For example, in some cases, the client device sends a participant identifier to designate the primary client device for the AR effect. The client device designated as the primary client device then renders an AR effect during the video call that differs from the AR effects rendered on other client devices.

[0118] For example, the primary client device can render the primary (or main) AR effect, while other client devices render secondary AR effects upon receiving the participant identifier and AR data of the AR effect via the AR data channel. In practice, in some embodiments, the primary AR effect can differ from the secondary AR effect. As an example, such as Figure 7 As shown, during the birthday-themed AR effect, the main client device 702b can render and display the main AR effect of the birthday message (via AR element 720), while other client devices 702a and 702c can render and display secondary AR effects of AR-based confetti via AR elements 715a and 715b.

[0119] In some instances, the client device receives AR data during a video call and renders the same AR effect. In some embodiments, the client device receives AR data and renders an AR effect with a uniform theme but slight variations in animation (e.g., a birthday theme where AR confetti on different client devices has different colors and sizes during the video call). Additionally, in some cases, the client device renders temporary AR effects during the video call (e.g., AR effects are rendered and displayed for a threshold amount of time before fading from the captured video).

[0120] In one or more embodiments, a client device sends AR data for an AR effect specific to a client device during a video call. In practice, the shared AR video call system 104 enables a client device to send an AR effect during a video call and allow another client device to render that AR effect independently (e.g., only a single client device or a selected number of client devices render the AR effect). For example, in some cases, the client device sends AR data for an AR effect along with participant identifiers (as described above) during a video call to allow one or more specific other client devices to render the AR effect.

[0121] Additionally, while one or more embodiments herein illustrate a shared AR video call system 104 implementing a birthday-themed AR effect during a video call, in one or more embodiments, the shared AR video call system 104 utilizes an AR data channel to implement various types of AR effects between multiple client devices in a video call. As an example, a client device can send AR data via an AR data channel (established by the shared AR video call system 104) so ​​that other client devices in the video call render an AR effect depicting snow within the captured video. In practice, the shared AR video call system 104 can implement a variety of AR effects, including, but not limited to, rain, confetti, fireworks, congratulatory roses, anniversary-themed AR effects, wedding-themed AR effects, graduation-themed AR effects, and / or holiday-themed AR effects.

[0122] Furthermore, in one or more embodiments, the shared AR video call system 104 can also implement AR effects that modify the appearance of one or more participant users within the captured video of one or more client devices located during a video call. For example, the shared AR video call system 104 can implement various AR effects, such as, but not limited to, hats, glasses, beards, clothing, attire, hairstyles, animated masks (e.g., face painting, masks, cartoon ears, cartoon eyes), and / or video capture style changes (e.g., cartoon effects on the video, comic book effects on the video, filters on the video). In practice, in one or more embodiments, the client device receives such AR effects and utilizes video capture capabilities on the client device (e.g., face tracking, graphics processor, motion tracking, ray tracing) to render and apply the AR effects within the captured video and / or on the participant users captured within the captured video.

[0123] In some embodiments, the shared AR video call system 104 also provides one or more optional AR options for display within a digital video call interface on a client device, for an AR environment, AR effects, AR-based activities, and / or individual AR elements. Specifically, upon receiving a selection of one or more optional AR options from the digital video call interface on the client device, the client device can send AR data to share and render the selected AR option across one or more client devices participating in the video call. In some cases, the shared AR video call system 104 can, in response to receiving a selection of one or more optional AR options from a client device, provide one or more AR elements (and / or other AR data) to one or more client devices in the video call to initialize the AR environment, AR effects, AR-based activities, and / or individual AR elements on the one or more client devices.

[0124] Furthermore, in some cases, one or more participating client devices among multiple participating client devices may send additional (or alternative) AR environments, AR effects, AR-based activities, and / or individual AR elements (e.g., via selection of alternative AR environments, AR effects, AR-based activities, and / or individual AR elements). In fact, in one or more embodiments, the shared AR video call system 104 enables multiple client devices in a video call to change or render new AR environments, AR effects, AR-based activities, and / or individual AR elements when AR data (or newly shared AR data) indicating the selection is detected within the AR data channel. Specifically, the shared AR video call system 104 enables multiple client devices to switch between various AR environments, various AR effects, various AR-based activities, and / or various individual AR elements during a video call by utilizing the AR data channel.

[0125] In some cases, during a video call, the shared AR video call system 104 can also provide one or more optional options for display within the digital video call interface of a client device. These options are used to select one or more primary participant devices for a shared AR environment, AR effects, AR-based activities, and / or individual AR elements. For example, during the selection of a shared AR environment, AR effects, AR-based activities, and / or individual AR elements, the client device can also receive selections for one or more primary users (or primary client devices). The client device can then transmit AR data via an AR data channel to share the AR environment, AR effects, AR-based activities, and / or individual AR elements using one or more participant identifiers of the selected one or more primary users. In practice, one or more receiving client devices in a video call can (as described above) use the participant identifiers to determine whether to render the AR environment, AR effects, AR-based activities, and / or individual AR elements, and / or to determine whether to render primary AR effects and secondary AR effects.

[0126] Furthermore, in one or more embodiments, the shared AR video call system 104 can implement audio components into a shared AR environment, AR effect, AR-based activity, and / or individual AR element during a video call. For example, client devices can also send audio information (or audio identifiers) via an AR data channel, causing client devices in the video call to play audio for the AR environment, AR effect, AR-based activity, and / or individual AR element (e.g., audio associated with the AR element). In some cases, the shared AR video call system 104 can provide a library of usable audio data for one or more AR environments, one or more AR effects, one or more AR-based activities, and / or one or more AR elements during a video call between multiple client devices.

[0127] Additionally, in one or more embodiments, the shared AR video call system 104 generates a graphical user interface to provide debugging tools for shared AR video calls. Specifically, in some embodiments, during a video call, the shared AR video call system 104 utilizes one or more simulated video call participants to provide optional options for previewing shared AR effects. In practice, the shared AR video call system 104 can provide optional options for selecting one or more simulated video call participants (e.g., participants depicted as video call participants, pre-recorded video call participants, or AI-based video call participants) for previewing the video call. Additionally, the shared AR video call system 104 can provide optional options for selecting a specific AR effect. Upon receiving a selection of an AR effect, the shared AR video call system 104 can provide a simulated preview of the AR effect displayed on one or more simulated video call participants. In some cases, the shared AR video call system 104 can also provide one or more optional options during a simulated video call with an AR effect preview for displaying AR data messages, sending AR data messages, and displaying AR data transmission history.

[0128] In one or more embodiments, the shared AR video calling system 104 is implemented on a computing device having various components and capabilities. One or more components of the shared AR video calling system 104 may include software, hardware, or both. For example, one or more components of the shared AR video calling system 104 may include one or more instructions stored on a computer-readable storage medium and executable by a processor of one or more computing devices (e.g., a client device or a server device). When executed by one or more processors, the computer-executable instructions of the shared AR video calling system 104 may cause one or more computing devices to perform the methods described herein. Alternatively, one or more components of the shared AR video calling system 104 may include hardware, such as a dedicated processing device that performs a function or a set of functions. Alternatively, one or more components of the shared AR video calling system 104 may include a combination of computer-executable instructions and hardware.

[0129] Furthermore, one or more components of the shared AR video calling system 104 can be implemented as one or more operating systems, as one or more standalone applications, as one or more modules of an application, as one or more plugins, as one or more library functions or functions that can be called by other applications, and / or as a cloud computing model. Therefore, one or more components of the shared AR video calling system 104 can be implemented as standalone applications, such as desktop or mobile applications. Additionally, one or more components of the shared AR video calling system 104 can be implemented as one or more web-based applications hosted on a remote server. Alternatively or additionally, one or more components of the shared AR video calling system 104 can be implemented within a set of mobile device applications or "apps".

[0130] Figures 1 to 7 The corresponding text and examples provide many different methods, systems, devices, and non-transitory computer-readable media for sharing the AR video calling system 104. In addition to the foregoing, such as Figure 8 and Figure 9 As shown, one or more embodiments can also be described based on flowcharts that include multiple actions for achieving a specific result. Figure 8 and Figure 9 It can be performed with more or fewer actions. Furthermore, Figure 8 and Figure 9 The multiple actions shown can be performed in different orders. Additionally, Figure 8 The described multiple actions and Figure 9 The described actions can be repeated or performed in parallel with each other, or performed in parallel with different instances of the same or similar actions.

[0131] For example, Figure 8 A flowchart of a series of actions 800 for rendering AR elements within a video call, according to one or more embodiments, is shown. Although Figure 8 Multiple actions according to one or more embodiments are illustrated, but alternative embodiments may omit, add, reorder, and / or modify them. Figure 8 Any of the multiple actions shown. In some implementations, Figure 8 Multiple actions are performed as part of a method. Alternatively, a non-transitory computer-readable medium may store multiple instructions thereon that, when executed by at least one processor, cause the computing device to perform... Figure 8 The actions described. In some embodiments, the system performs... Figure 8 The system performs actions within the system. For example, in one or more embodiments, the system includes at least one processor. The system may also include a non-transitory computer-readable medium comprising a plurality of instructions that, when executed by the at least one processor, cause the system to perform... Figure 8 The actions within.

[0132] like Figure 8 As shown, a series of actions 800 includes action 810 of conducting a video call with a receiving participant device. Specifically, action 810 may include a client device conducting a video call with the receiving participant device by receiving video data from the receiving participant device via a video data channel established for the video call.

[0133] Furthermore, a series of actions 800 includes action 820 of displaying video within the digital video call interface. Specifically, action 820 may include displaying a first video captured by a client device within the digital video call interface. Additionally, action 820 may include displaying a second video from the receiving participant's device within the digital video call interface by rendering video data received via a video data channel. Furthermore, action 820 may include displaying a second video from the receiving participant's device within the digital video call interface, depicting interactions with the receiving user portrayed in the second video. Additionally, action 820 may include detecting user movement interactions within the first video captured by the client device and modifying augmented reality elements within the first video based on these user movement interactions.

[0134] A series of actions 800 includes action 830 of receiving augmented reality data via an augmented reality data channel. Specifically, action 830 may include receiving augmented reality data indicating augmented reality elements from a receiving participant's device via an augmented reality data channel established for a video call with the receiving participant's device. Additionally, action 830 may include receiving supplemental augmented reality data indicating modifications to augmented reality elements from the receiving participant's device via the augmented reality data channel.

[0135] Furthermore, action 830 may include identifying a participant identifier from the additional augmented reality data. Additionally, action 830 may include identifying an object vector from the additional augmented reality data. Furthermore, action 830 may include identifying within the augmented reality data an indication of a client device's selection as a primary participant. Additionally, action 830 may include receiving augmented reality data indicating augmented reality elements by filtering a set of augmented reality data in the augmented reality data channel.

[0136] Furthermore, the series of actions 800 includes action 840 of rendering augmented reality elements. Specifically, action 840 may include rendering augmented reality elements within a first video displayed within the digital video call interface based on received augmented reality data. Additionally, action 840 may include rendering augmented reality elements within the first video displayed within the digital video call interface, for sharing an augmented reality environment scene depicted in a second video from the receiving participant's device. For example, the augmented reality elements may modify at least one of the background, foreground, or video filters of the first video. Furthermore, action 840 may include rendering augmented reality elements within the first video for displaying augmented reality effects of user interaction depicted in the first video. Additionally, action 840 may include rendering augmented reality effects on the client device within the first video, different from the additional augmented reality effects displayed in the second video, based on the client device's selection to indicate the main participant.

[0137] Additionally, action 840 may include modifying augmented reality elements within the first video based on augmented reality data. In some cases, action 840 includes: displaying a second video, depicting an augmented reality element exiting the second video, from the receiving participant's device within a digital video call interface by rendering video data received via a video data channel; and rendering augmented reality elements within the first video depicting an augmented reality element entering the first video based on the received augmented reality data. Furthermore, action 840 may include: determining which augmented reality elements to render within the first video by matching a participant identifier with a client device. Additionally, action 840 may include rendering the augmented reality element at a specific location within the first video using an object vector and moving the augmented reality element along a specific direction.

[0138] Now transferred to Figure 9 , Figure 9 A flowchart of a series of actions 900 for establishing a shared augmented reality video call, according to one or more embodiments, is shown. Although Figure 9 Multiple actions according to one or more embodiments are illustrated, but alternative embodiments may omit, add, reorder, and / or modify them. Figure 9 Any of the multiple actions shown. In some implementations, Figure 9 Multiple actions are performed as part of a method. Alternatively, a non-transitory computer-readable medium may store multiple instructions thereon that, when executed by at least one processor, cause the computing device to perform... Figure 9 The actions described. In some embodiments, the system performs... Figure 9 The system performs actions within the system. For example, in one or more embodiments, the system includes at least one processor. The system may also include a non-transitory computer-readable medium comprising a plurality of instructions that, when executed by the at least one processor, cause the system to perform... Figure 9 The actions within.

[0139] like Figure 9 As shown, a series of actions 900 includes action 910 of establishing a shared augmented reality video call among multiple participant devices. In particular, action 910 may include establishing a shared augmented reality video call between the first participant device and the second participant device by establishing a video data channel and an augmented reality data channel between the first participant device and the second participant device.

[0140] The series of actions 900 also includes the action 920 of the first participant device and the second participant device sending video. Specifically, action 920 may include the first participant device acquiring a first video stream and sending the first video stream to the second participant device via a video data channel. Furthermore, action 920 may include the second participant device acquiring a second video stream and sending the second video stream to the first participant device via a video data channel. Action 920 may also include the first participant device sending a first video stream depicting augmented reality elements within the first video stream to the second participant device via a video data channel.

[0141] Furthermore, a series of actions 900 includes action 930 where the first participant device sends an augmented reality identifier via the augmented reality data channel. Specifically, action 930 may include the first participant device sending the augmented reality identifier to a second participant device via the augmented reality data channel. Additionally, action 930 may include the first participant device sending an additional augmented reality identifier indicating interaction with an augmented reality element. Furthermore, action 930 may include the first participant device sending an augmented reality identifier indicating an object vector of the augmented reality element to the second participant device via the augmented reality data channel. Additionally, action 930 may include the first participant device sending augmented reality identifiers to multiple participant devices via the augmented reality data channel.

[0142] The series of actions 900 also includes action 940 of the second participant device rendering augmented reality elements. Specifically, action 940 may include the second participant device rendering a second video stream using augmented reality elements associated with an augmented reality identifier. Furthermore, action 940 may include: the first participant device rendering a first video stream using additional augmented reality elements associated with an augmented reality identifier to share the augmented reality environment scene depicted within the second video stream from the second participant device. Additionally, action 940 may include the second participant device rendering the second video stream using modifications to the augmented reality elements based on the additional augmented reality identifier.

[0143] Additionally, action 940 may include: once the augmented reality identifier has been sent to the second participant device via the augmented reality data channel, the first participant device sends a first video stream to the second participant device via the video data channel, depicting an augmented reality element exiting the first video stream. Furthermore, action 940 may include: once the first participant device sends the augmented reality identifier to the second participant device via the augmented reality data channel, the second participant device renders the second video stream using the augmented reality element associated with the augmented reality identifier that is entering the second video stream. Additionally, action 940 may include the second participant device using an object vector to render the second video stream using an augmented reality element associated with the augmented reality identifier that is located at a specific position and moves in a specific direction.

[0144] Furthermore, action 940 may include a second participant device rendering the second video stream using augmented reality elements for displaying augmented reality effects of user interactions depicted within the second video stream. Additionally, action 940 may include a first participant device rendering the first video stream using additional augmented reality effects of user interactions depicted within the first video stream. Furthermore, action 940 may include multiple participant devices rendering the video stream using augmented reality elements associated with augmented reality identifiers.

[0145] Embodiments of this disclosure may include or utilize a dedicated or general-purpose computer including computer hardware (e.g., one or more processors and system memory), as discussed in more detail below. Embodiments within the scope of this disclosure also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. In particular, one or more of the processes described herein may be implemented at least in part as a plurality of instructions embodied in a non-transitory computer-readable medium and executable by one or more computing devices (e.g., any of the media content access devices described herein). Generally, a processor (e.g., a microprocessor) receives a plurality of instructions from a non-transitory computer-readable medium (e.g., memory) and executes those instructions to perform one or more processes, including one or more of the processes described herein.

[0146] Computer-readable media can be any available medium that can be accessed by general-purpose or special-purpose computer systems. A computer-readable medium storing computer-executable instructions is a non-transitory computer-readable storage medium (device). A computer-readable medium carrying computer-executable instructions is a transmission medium. Therefore, by way of example and not limitation, embodiments of this disclosure may include at least two distinctly different kinds of computer-readable media: non-transitory computer-readable storage media (devices) and transmission media.

[0147] Non-transitory computer-readable storage media (devices) include random-access memory (RAM), read-only memory (ROM), electronically erasable programmable read-only memory (EEPROM), optical disc read-only memory (CD-ROM), solid-state drive (SSD) (e.g., RAM-based), flash memory, phase-change memory (PCM), other types of memory, other optical disc storage, disk storage or other magnetic storage devices, or any other medium that can be used to store desired program code tools in the form of computer-executable instructions or data structures and is accessible by a general-purpose or special-purpose computer.

[0148] A “network” is defined as one or more data links that enable the transmission of electronic data between multiple computer systems and / or multiple modules and / or other electronic devices. When information is transmitted or provided to a computer via a network or another communication connection (either hardwired, wireless, or a combination of hardwired and wireless), the computer appropriately regards that connection as a transmission medium. Transmission media may include networks and / or data links that can be used to transmit desired program code tools in the form of computer-executable instructions or data structures, and that can be accessed by general-purpose or special-purpose computers. Various combinations of the above should also be included within the scope of computer-readable media.

[0149] Furthermore, program code tools in the form of computer-executable instructions or data structures can be automatically transferred from a transmission medium to a non-transitory computer-readable storage medium (device) (or automatically transferred from a non-transitory computer-readable storage medium to a transmission medium) upon arrival at various computer system components. For example, computer-executable instructions or data structures received via a network or data link can be cached in RAM within a network interface module (e.g., a "network interface controller (NIC)") and then ultimately transferred to the computer system RAM and / or ultimately to a less volatile computer storage medium (device) at the computer system. Therefore, it should be understood that a non-transitory computer-readable storage medium (device) can be included in computer system components that also (or even primarily) utilize the transmission medium.

[0150] Computer-executable instructions, for example, comprise multiple instructions and data that, when executed by a processor, cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a function or a set of functions. In some embodiments, the computer-executable instructions are executed by a general-purpose computer to turn that general-purpose computer into a special-purpose computer that implements the elements of this disclosure. Computer-executable instructions may be, for example, binary code, intermediate format instructions such as assembly language, or even source code. Although the subject matter has been described in language relating to structural features and / or methodological actions, it will be understood that the subject matter defined in the appended claims is not necessarily limited to the features or actions described above. Rather, the described features and actions are disclosed as exemplary forms of implementing the claims.

[0151] Those skilled in the art will recognize that this disclosure can be practiced in a variety of networked computing environments with many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, networked personal computers (PCs), minicomputers, mainframes, mobile phones, personal digital assistants (PDAs), tablet computers, pagers, routers, and switches. This disclosure can also be practiced in distributed system environments where local and remote computer systems connected via a network (either via hardwired data links, wireless data links, or a combination of hardwired and wireless data links) perform tasks. In a distributed system environment, program modules can reside on both local and remote memory storage devices.

[0152] Embodiments of this disclosure can also be implemented in a cloud computing environment. As used herein, the term "cloud computing" refers to a model for enabling on-demand network access to a shared pool of configurable computing resources. For example, cloud computing can be employed in a market environment to provide ubiquitous and convenient on-demand access to a shared pool of configurable computing resources. The shared pool of configurable computing resources can be rapidly provisioned via virtualization and deployed with low management effort or service provision interaction, and then scaled up accordingly.

[0153] Cloud computing models can be composed of various features, such as on-demand self-service, broadband access, resource pooling, rapid elasticity, and measurement services. They can also utilize various service models, such as Software as a Service (SaaS), Platform as a Service (PaaS), and Infrastructure as a Service (IaaS). Cloud computing models can also be deployed using different deployment models, such as private clouds, community clouds, public clouds, and hybrid clouds. Furthermore, as used herein, the term "cloud computing environment" refers to an environment in which cloud computing is employed.

[0154] Figure 10 A block diagram of an example computing device 1000 is shown, which can be configured to perform one or more of the processes described above. It will be appreciated that one or more computing devices (e.g., computing device 1000) can represent the computing devices described above (e.g., one or more server devices 102, and / or client devices 108a and 108b-108n). In one or more embodiments, computing device 1000 can be a mobile device (e.g., mobile phone, smartphone, PDA, tablet, laptop, camera, tracker, watch, wearable device, head-mounted display, etc.). In some embodiments, computing device 1000 can be a non-mobile device (e.g., desktop computer, or another type of client device). Furthermore, computing device 1000 can be a server device including cloud-based processing and storage capabilities.

[0155] like Figure 10 As shown, computing device 1000 may include one or more processors 1002, memory 1004, storage device 1006, input / output interface 1008 (or "I / O interface 1008"), and communication interface 1010 that can be communicatively coupled via a communication infrastructure (e.g., bus 1012). Although in Figure 10 The image shows a computing device 1000, but... Figure 10 The multiple components shown are not intended to be limiting. In other embodiments, additional or alternative components may be used. Furthermore, in some embodiments, the computing device 1000 includes more than... Figure 10 The components shown have fewer parts. A more detailed description will follow. Figure 10 The computing device 1000 shown has multiple components.

[0156] In a particular embodiment, the one or more processors 1002 include hardware for executing multiple instructions (e.g., those that constitute a computer program). By way of example and not limitation, in order to execute multiple instructions, the one or more processors 1002 may retrieve (or fetch) multiple instructions from internal registers, internal cache memory, memory 1004, or storage device 1006 and decode and execute those instructions.

[0157] Computing device 1000 includes memory 1004 coupled to one or more processors 1002. Memory 1004 can be used to store data, metadata, and programs executable by one or more processors. Memory 1004 can include one or more of volatile and non-volatile memory, such as random access memory (“RAM”), read-only memory (“ROM”), solid-state drive (“SSD”), flash memory, phase-change memory (“PCM”), or other types of data storage. Memory 1004 can be internal memory or distributed memory.

[0158] Computing device 1000 includes storage device 1006, which includes memory for storing data or instructions. By way of example and not limitation, storage device 1006 may include the non-transitory storage media described above. Storage device 1006 may include a hard disk drive (HDD), flash memory, a Universal Serial Bus (USB) drive, or a combination of these or other storage devices.

[0159] As shown in the figure, computing device 1000 includes one or more I / O interfaces 1008 provided to allow a user to provide input to computing device 1000 (e.g., by touch), receive output from computing device 1000, and otherwise transmit data to and from computing device 1000. These I / O interfaces 1008 may include a mouse, keypad or keyboard, touchscreen, camera, optical scanner, network interface, modem, other known I / O devices, or combinations of these I / O interfaces 1008. Touchscreens can be activated with a stylus or finger.

[0160] I / O interface 1008 may include one or more devices for presenting output to a user, including but not limited to a graphics engine, a display (e.g., a screen), one or more output drivers (e.g., display drivers), one or more audio speakers, and one or more audio drivers. In some embodiments, I / O interface 1008 is configured to provide graphical data to the display for presentation to a user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content that may be served in a particular implementation.

[0161] The computing device 1000 may also include a communication interface 1010. The communication interface 1010 may include hardware, software, or both. The communication interface 1010 provides one or more interfaces for communication (e.g., packet-based communication) between the computing device and one or more other computing devices or one or more networks. By way of example and not by way of limitation, the communication interface 1010 may include a network interface controller (NIC) or network adapter for communicating with Ethernet or other wired networks, or a wireless network interface controller (wireless NIC, WNIC) or wireless adapter for communicating with wireless networks (e.g., Wi-Fi). The computing device 1000 may also include a bus 1012. The bus 1012 may include hardware, software, or both for connecting multiple components of the computing device 1000 to each other. As an example, the bus 1012 may include one or more types of buses.

[0162] Embodiments of the present invention may include an artificial reality system or a combination thereof. Artificial reality is a form of reality that has been adjusted in some way before being presented to a user. This artificial reality may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or generated content combined with captured content (e.g., real-world photographs). Artificial reality content may include video, audio, haptic feedback, or some combination thereof, and any of the above may be presented in a single channel or in multiple channels (e.g., stereoscopic video that produces a three-dimensional effect for the viewer). Additionally, in some embodiments, artificial reality may be associated with applications, products, accessories, services, or some combination thereof, which are used, for example, to create content in and / or use in the artificial reality (e.g., to perform activities in the artificial reality). Artificial reality systems that deliver artificial reality content can be implemented on a variety of platforms, including head-mounted displays (HMDs) connected to a host computer system, standalone HMDs, mobile devices or computing systems, or any other hardware platform capable of delivering artificial reality content to one or more viewers.

[0163] As mentioned above, communication systems can be included in social networking systems. A social networking system enables multiple users (e.g., individuals or organizations) to interact with the system and with each other. Social networking systems can utilize user input to create and store user profiles associated with each user. As described above, user profiles can include demographic information, communication channel information, and information about the user's personal interests.

[0164] More specifically, user profile information may include, for example, biometric information, demographic information, behavioral information, social information, or other types of descriptive information, such as work experience, educational history, hobbies or preferences, interests, intimacy, or location. Interest information may include interests associated with one or more categories, which may be general or specific. As an example, if a user “likes” items about shoe brands, then that category could be brands.

[0165] Social networking systems can also utilize user input to create and store records of a user's relationships with other users on the network, and provide services to facilitate social interaction between or among users (e.g., wall posts, photo sharing, online calendars and event organization, messaging, games, or advertising). Furthermore, social networking systems can allow users to post photos and other multimedia content to their profile page (often referred to as a "wall post" or "timeline post") or albums, both of which are accessible to other users on the network, depending on the user's privacy settings. In this document, the term "friend" can refer to any other user on the social networking system with whom the user has formed a connection, association, or relationship through the social networking system.

[0166] Figure 11 An example network environment 1100 for a social networking system is shown. Network environment 1100 includes client devices 1106, networking systems 1102 (e.g., social networking systems and / or electronic messaging systems), and third-party systems 1108 connected to each other via network 1104. Although Figure 11 A specific arrangement of client device 1106, networking system 1102, third-party system 1108, and network 1104 is shown, but this disclosure contemplates any suitable arrangement of client device 1106, networking system 1102, third-party system 1108, and network 1104. By way of example and not by limitation, two or more of client device 1106, networking system 1102, and third-party system 1108 may bypass network 1104 and connect directly to each other. As another example, two or more of client device 1106, networking system 1102, and third-party system 1108 may be physically or logically located, wholly or partially, in the same location. Furthermore, although... Figure 11 A specific number of client devices 1106, networking systems 1102, third-party systems 1108, and networks 1104 are shown, but this disclosure contemplates any suitable number of client devices 1106, networking systems 1102, third-party systems 1108, and networks 1104. As an example and not by limitation, network environment 1100 may include multiple client devices 1106, multiple networking systems 1102, multiple third-party systems 1108, and multiple networks 1104.

[0167] This disclosure considers any suitable network 1104. By way of example and not by limitation, one or more portions of network 1104 may include an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless wide area network (WWAN), metropolitan area network (MAN), a portion of the Internet, a portion of the Public Switched Telephone Network (PSTN), a cellular telephone network, or a combination of two or more of these networks. Network 1104 may include one or more networks 1104.

[0168] Links can connect client device 1106, networking system 1102, and third-party system 1108 to communication network 1104 or to each other. This disclosure considers any suitable link. In a particular embodiment, one or more links include one or more wired links (e.g., Digital Subscriber Line (DSL) or Data Over Cable Service Interface Specification (DOCSIS)), wireless links (e.g., Wi-Fi or Worldwide Interoperability for Microwave Access (WiMAX)), or optical links (e.g., Synchronous Optical Network (SONET) or Synchronous Digital Hierarchy (SDH)). In a particular embodiment, one or more links each include an ad hoc network, intranet, extranet, VPN, LAN, WLAN, WAN, WWAN, MAN, a portion of the Internet, a portion of the PSTN, a cellular-based network, a satellite-based network, another link, or a combination of two or more of these links. Multiple links are not necessarily identical throughout the network environment 1100. In one or more respects, one or more first links may differ from one or more second links.

[0169] In a particular embodiment, client device 1106 may be an electronic device comprising hardware, software, or embedded logic components, or a combination of two or more such components, and the electronic device is capable of performing appropriate functions implemented or supported by client device 1106. By way of example and not limitation, client device 1106 may include a computer system, such as an augmented reality display device, desktop computer, notebook or laptop computer, netbook, tablet computer, e-book reader, GPS device, camera, personal digital assistant (PDA), handheld electronic device, cellular phone, smartphone, other suitable electronic device, or any suitable combination thereof. This disclosure contemplates any suitable client device 1106. Client device 1106 enables network users at client device 1106 to access network 1104. Client device 1106 enables its users to communicate with other users at other client devices 1106.

[0170] In a particular embodiment, client device 1106 may include a web browser and may have one or more add-ons, plugins, or other extensions. A user at client device 1106 may access a Uniform Resource Locator (URL) or another address that directs the web browser to a specific server (e.g., a server or a server associated with third-party system 1108), and the web browser may generate a Hypertext Transfer Protocol (HTTP) request and send the HTTP request to the server. The server may receive the HTTP request and, in response, send one or more Hypertext Markup Language (HTML) files to client device 1106. Client device 1106 may render the webpage based on the HTML file from the server for presentation to the user. This disclosure contemplates any suitable webpage file. By way of example and not limitation, webpages may be rendered based on HTML files, Extensible Hypertext Markup Language (XHTML) files, or Extensible Markup Language (XML) files as needed. These pages can also execute scripts, such as, but not limited to, those written in JavaScript, Java, Microsoft Silverlight, and combinations of markup languages ​​and scripts such as AJAX (asynchronous JavaScript and XML). In this document, where appropriate, a reference to a webpage includes one or more corresponding webpage files (which the browser can use to render the webpage), and vice versa.

[0171] In a particular embodiment, the networking system 1102 may be a network-addressable computing system capable of hosting online social networks. The networking system 1102 may generate, store, receive, and transmit social network data, such as user profile data, concept-profile data, social graph information, or other suitable data related to the online social network. The networking system 1102 may be accessed either directly by other components in the network environment 1100 or via network 1104 by other components in the network environment 1100. In a particular embodiment, the networking system 1102 may include one or more servers. Each server may be a single server or a distributed server spanning multiple computers or multiple data centers. Servers may be of various types, such as, but not limited to, web servers, news servers, mail servers, messaging servers, advertising servers, file servers, application servers, exchange servers, database servers, proxy servers, another server adapted to perform the functions or processes described herein, or any combination thereof. In a particular embodiment, each server may include hardware, software, or embedded logic components, or combinations of two or more such components, for performing appropriate functions implemented or supported by the server. In a particular embodiment, the networking system 1102 may include one or more data repositories. The data repository can be used to store various types of information. In a particular embodiment, the information stored in the data repository can be organized according to a specific data structure. In a particular embodiment, each data repository may be a relational database, a columnar database, a correlated database, or other suitable database. Although this disclosure describes or illustrates specific types of databases, this disclosure contemplates any suitable type of database. Specific embodiments may provide an interface that enables client device 1106, networking system 1102, or third-party system 1108 to manage, retrieve, modify, add, or delete information stored in the data repository.

[0172] In a particular embodiment, the networking system 1102 may store one or more social graphs in one or more data repositories. In a particular embodiment, the social graph may include multiple nodes—which may include multiple user nodes (each user node corresponds to a specific user) or multiple concept nodes (each concept node corresponds to a specific concept)—and multiple edges connecting these nodes. The networking system 1102 may provide users of the online social network with the ability to communicate and interact with other users. In a particular embodiment, a user may join an online social network via the networking system 1102 and then add connections (e.g., relationships) to multiple other users they wish to connect to within the networking system 1102. In this document, the term "friend" may refer to any other user with whom a user has already formed a connection, association, or relationship via the networking system 1102.

[0173] In a particular embodiment, networking system 1102 may provide users with the ability to take action on various types of items or objects supported by networking system 1102. By way of example and not limitation, items and objects may include groups or social networks to which a user of networking system 1102 may belong, events or calendar entries that a user may be interested in, computer-based applications that a user can use, transactions that allow a user to buy or sell items through services, user-executable interactions with advertisements, or other suitable items or objects. Users may interact with anything that can be displayed in networking system 1102 or by an external system of third-party system 1108, which is separate from networking system 1102 and coupled to networking system 1102 via network 1104.

[0174] In certain embodiments, the networking system 1102 may be able to link various entities. By way of example and not limitation, the networking system 1102 may enable multiple users to interact with each other and receive content from third-party systems 1108 or other entities, or allow users to interact with these entities through application programming interfaces (APIs) or other communication channels.

[0175] In a particular embodiment, third-party system 1108 may include one or more types of servers, one or more data repositories, one or more interfaces, including but not limited to APIs, one or more web services, one or more content sources, one or more networks, or any other suitable component (e.g., with which the server can communicate). Third-party system 1108 may be operated by an entity different from the entity operating network system 1102. However, in a particular embodiment, network system 1102 and third-party system 1108 may operate in combination to provide social networking services to users of either network system 1102 or third-party system 1108. In this sense, network system 1102 may provide a platform or backbone that other systems (e.g., third-party system 1108) can use to provide social networking services and functionality to users on the Internet.

[0176] In a particular embodiment, third-party system 1108 may include a third-party content object provider. The third-party content object provider may include one or more sources of content objects that can be transmitted to client device 1106. By way of example, and not limitation, the content object may include information about things or activities of interest to the user, such as movie showtimes, movie reviews, restaurant reviews, restaurant menus, product information and reviews, or other suitable information. By way of another example, and not limitation, the content object may include incentive content objects, such as coupons, discount vouchers, gift certificates, or other suitable incentive content objects.

[0177] In a particular embodiment, the networking system 1102 also includes a plurality of user-generated content objects that can enhance user interaction with the networking system 1102. User-generated content objects can include any content that a user can add, upload, send, or "post" to the networking system 1102. As an example, and not by way of limitation, a user transmits a post from client device 1106 to the networking system 1102. A post can include data such as status updates or other text data, location information, photos, videos, links, music, or other similar data or media. Content can also be added to the networking system 1102 by a third-party system 1108 via a "communication channel" (e.g., a news feed or stream).

[0178] In a particular embodiment, the networked system 1102 may include various servers, subsystems, programs, modules, logs, and data repositories. In a particular embodiment, the networked system 1102 may include one or more of the following: a web server, an action logger, an API request server, a relevance-and-rank engine, a content object classifier, a notification controller, an action log, a third-party content object exposure log, an inference module, an authorization / privacy server, a search module, an advertising-targeting module, a user interface module, a user profile repository, a connection repository, a third-party content repository, or a location repository. The networked system 1102 may also include suitable components such as network interfaces, security mechanisms, load balancers, failover servers, management and network operation consoles, other suitable components, or any suitable combination thereof. In a particular embodiment, the networked system 1102 may include one or more user profile repositories for storing user profiles. User profiles may include, for example, biometric information, demographic information, behavioral information, social information, or other types of descriptive information, such as work experience, educational history, hobbies or preferences, interests, intimacy, or location. Interest information may include interests associated with one or more categories. These categories may be general or specific. As an example, and not by limitation, if a user “likes” items related to a brand of shoes, the category could be the brand, or the general category of “shoes” or “clothing.” A connection repository may be used to store connection information about a user. Connection information may indicate users with similar or shared work experience, group memberships, hobbies, educational history, or users who are associated with or share common attributes in any way. Connection information may also include user-defined connections between different users and content (both internal and external). A web server may be used to link the networked system 1102 to one or more client devices 1106 or one or more third-party systems 1108 via network 1104. The web server may include a mail server, or other messaging functionality for receiving and routing messages between the networked system 1102 and one or more client devices 1106. The API request server allows a third-party system 1108 to access information from the networked system 1102 by calling one or more APIs.An operation logger can be used to receive information from the web server regarding a user's actions of opening or closing the network system 1102. Combined with the operation log, a log of user-published third-party content objects can be maintained. A notification controller can provide information about content objects to client device 1106. This information can be pushed to client device 1106 as a notification, or retrieved from client device 1106 in response to a request received from client device 1106.

[0179] An authorization server can be used to enforce privacy settings for one or more users on networked system 1102. User privacy settings determine how specific information associated with a user can be shared. For example, the authorization server can allow users to choose whether or not networked system 1102 can record their actions or share their actions with other systems (e.g., third-party system 1108) by setting appropriate privacy settings. A third-party content object repository can be used to store content objects received from third parties (e.g., third-party system 1108). A location repository can be used to store location information received from client device 1106 associated with a user. An advertising pricing module can combine social information, current time, location information, or other suitable information to provide relevant advertisements to users in the form of notifications.

[0180] Figure 12 An example social graph 1200 is illustrated. In a particular embodiment, the networking system 1102 may store one or more social graphs 1200 in one or more data repositories. In a particular embodiment, the social graph 1200 may include multiple nodes—which may include multiple user nodes 1202 or multiple concept nodes 1204—and multiple edges 1206 connecting these nodes. For educational purposes, Figure 12 The example social graph 1200 shown is displayed in a two-dimensional visual graph representation. In a particular embodiment, a networked system 1102, a client device 1106, or a third-party system 1108 can access the social graph 1200 and associated social graph information for suitable applications. Multiple nodes and edges of the social graph 1200 can be stored as data objects in a data repository (e.g., a social graph database). Such a data repository may include one or more searchable or queryable indexes of the multiple nodes or edges of the social graph 1200.

[0181] In a particular embodiment, user node 1202 may correspond to a user of networked system 1102. As an example and not by limitation, a user may be an individual (human user), an entity (e.g., a business, company, or third-party application), or a group (e.g., a group of individuals or entities) that interacts or communicates with or through networked system 1102. In a particular embodiment, when a user registers an account on networked system 1102, networked system 1102 may create a user node 1202 corresponding to that user and store the user node 1202 in one or more data repositories. Where appropriate, the user and user node 1202 described herein may refer to a registered user and the user node 1202 associated with the registered user. Additionally or alternatively, where appropriate, the user and user node 1202 described herein may refer to a user who has not registered on networked system 1102. In a particular embodiment, user node 1202 may be associated with information provided by the user or information collected by various systems, including networked system 1102. As an example, and not as a limitation, a user may provide his or her name, profile picture, contact information, date of birth, gender, marital status, family status, occupation, educational background, preferences, interests, or other demographic information. In a particular embodiment, user node 1202 may be associated with one or more data objects corresponding to information associated with the user. In a particular embodiment, user node 1202 may correspond to one or more web pages.

[0182] In a particular embodiment, concept node 1204 may correspond to a concept. As an example and not by limitation, a concept may correspond to a location (e.g., a movie theater, restaurant, landmark, or city), a website (e.g., a website associated with network system 1102, or a third-party website associated with a web application server), an entity (e.g., a person, company, group, sports team, or celebrity), a resource (e.g., audio files, video files, digital photographs, text files, structured documents, or applications) that may be located on a server (e.g., a web application server) within or outside network system 1102, or to real estate or intellectual property (e.g., sculptures, paintings, films, games, songs, ideas, photographs, or written works), a game, an event, an idea or theory, another suitable concept, or two or more such concepts. Concept node 1204 may be associated with information about a concept provided by a user or information collected by various systems (including network system 1102). By way of example and not limitation, concept information may include a name or title, one or more images (e.g., an image of a book cover), location (e.g., an address or geographic location), website (which may be associated with a URL), contact information (e.g., a phone number or email address), other suitable concept information, or any suitable combination of such information. In a particular embodiment, concept node 1204 may be associated with one or more data objects corresponding to information associated with concept node 1204. In a particular embodiment, concept node 1204 may correspond to one or more web pages.

[0183] In a particular embodiment, nodes in the social graph 1200 may represent web pages (which may be referred to as “profile pages”) or be represented by web pages. Profile pages may be hosted or accessed by the networking system 1102. Profile pages may also be hosted on a third-party website associated with the third-party system 1108. As an example, and not by limitation, a profile page corresponding to a particular external web page may be that particular external web page, and that profile page may correspond to a particular concept node 1204. Profile pages may be viewed by all other users or a selected subset of other users. As an example, and not by limitation, user node 1202 may have a corresponding user profile page, in which the corresponding user user can add content, make statements, or otherwise express himself or her. As another example, and not by limitation, concept node 1204 may have a corresponding concept profile page, in which one or more users can add content, make statements, or express themselves, particularly in relation to the concept corresponding to concept node 1204.

[0184] In a particular embodiment, concept node 1204 may represent a third-party webpage or resource hosted by third-party system 1108. The third-party webpage or resource may include content among other elements, optional icons or other icons, or other interactive objects representing actions or activities (which may be implemented, for example, using JavaScript, AJAX, or PHP code). As an example and not by limitation, the third-party webpage may include optional icons such as "like," "check-in," "eat," "recommend," or other suitable actions or activities. A user viewing the third-party webpage can perform an action by selecting one of several icons (e.g., "eat"), causing client device 1106 to send a message instructing the user's action to network system 1102. In response to this message, network system 1102 may create an edge (e.g., an "eat" edge) between user node 1202 corresponding to the user and concept node 1204 corresponding to the third-party webpage or resource, and store edge 1206 in one or more data repositories.

[0185] In a particular embodiment, a pair of nodes in the social graph 1200 can be connected to each other via one or more edges 1206. An edge 1206 connecting a pair of nodes can represent a relationship between the two nodes. In a particular embodiment, an edge 1206 can include or represent one or more data objects or attributes corresponding to a relationship between a pair of nodes. As an example, and not by way of limitation, a first user can indicate that a second user is a "friend" of the first user. In response to this indication, the networking system 1102 can send a "friend request" to the second user. If the second user confirms the "friend request," the networking system 1102 can create an edge 1206 in the social graph 1200 connecting the first user's user node 1202 to the second user's user node 1202, and store this edge 1206 as social graph information in one or more data repositories among multiple data repositories. Figure 12In the example, social graph 1200 includes edges 1206 indicating a friendship relationship between user nodes 1202 of user "A" and user "B", and edges indicating a friendship relationship between user nodes 1202 of user "C" and user "B". Although this disclosure describes or illustrates a particular edge 1206 having a particular attribute connecting a particular user node 1202, this disclosure considers any suitable edge 1206 having any suitable attribute connecting user nodes 1202. As an example and not by limitation, edge 1206 may represent a friendship, family relationship, business or employment relationship, fan relationship, follower relationship, visitor relationship, subscriber relationship, superior / subordinate relationship, reciprocal relationship, non-reciprocal relationship, another suitable type of relationship, or two or more of such relationships. Furthermore, although this disclosure generally describes nodes as being connected, this disclosure also describes users or concepts as being connected. In this paper, where appropriate, a reference to a user or concept being connected may refer to the node corresponding to those users or concepts being connected in the social graph 1200 by one or more edges 1206.

[0186] In a particular embodiment, edge 1206 between user node 1202 and concept node 1204 may represent a specific action or activity performed by a user associated with user node 1202 to a concept associated with concept node 1204. This is intended as an example and not as a limitation. Figure 12 As shown, users can "like," "regularly visit," "play," "listen," "cook," "work on," or "watch," each of which can correspond to an edge type or subtype. The concept information page corresponding to concept node 1204 may, for example, include an optional "check-in" icon (e.g., a clickable "check-in" icon) or an optional "add to favorites" icon. Similarly, after a user clicks these icons, the network system 1102 can create a "like" edge or a "check-in" edge in response to the user action corresponding to the respective action. As another example, rather than by restriction, a user (user "C") can use a specific application (MUSIC, which is an online music application) to listen to a specific song ("Ramble On"). In this case, the network system 1102 can create a "listen" edge 1206 and a "use" edge (e.g., ...) between the user node 1202 corresponding to the user and the concept node 1204 corresponding to the song and application. Figure 12 As shown), to instruct the user to listen to the song and use the application. Furthermore, the network system 1102 can create "playback" edges 1206 (such as...) between multiple conceptual nodes 1204 corresponding to the song and the application. Figure 12As shown), to indicate that a specific application has played a specific song. In this case, the "play" edge 1206 corresponds to an action performed by an external application (MUSIC) on an external audio file (song "Imagine"). Although this disclosure describes a specific edge 1206 with a specific attribute connecting user node 1202 and concept node 1204, this disclosure considers any suitable edge 1206 with any suitable attribute connecting user node 1202 and concept node 1204. Furthermore, although this disclosure describes an edge representing a single relationship between user node 1202 and concept node 1204, this disclosure considers an edge representing one or more relationships between user node 1202 and concept node 1204. As an example and not by limitation, edge 1206 could represent both a user liking a specific concept and a user having used a specific concept. Alternatively, another edge 1206 could represent a relationship between user node 1202 and concept node 1204 (such as...). Figure 12 The number of each type of relationship (or multiples of a single relationship) between user node 1202 of user “E” and concept node 1204 of “MUSIC”.

[0187] In a particular embodiment, the networking system 1102 may create an edge 1206 between user nodes 1202 and concept nodes 1204 in the social graph 1200. As an example, and not by limitation, a user viewing a concept profile page (e.g., using a web browser or dedicated application hosted by the user's client device 1106) may indicate that he or she likes a concept represented by concept node 1204 by clicking or selecting a "like" icon. This may cause the user's client device 1106 to send a message to the networking system 1102 indicating that the user likes the concept associated with the concept profile page. In response to this message, the networking system 1102 may create an edge 1206 between the user node 1202 associated with the user and concept node 1204, as shown by the "like" edge 1206 between the user node and concept node 1204. In a particular embodiment, the networking system 1102 may store the edge 1206 in one or more data repositories. In a particular embodiment, the edge 1206 may be automatically formed by the networking system 1102 in response to a specific user action. As an example, and not as a limitation, if a first user uploads a picture, watches a movie, or listens to music, an edge 1206 can be formed between the user node 1202 corresponding to the first user and the concept node 1204 corresponding to these concepts. Although this disclosure describes the formation of a particular edge 1206 in a specific manner, this disclosure contemplates the formation of any suitable edge 1206 in any suitable manner.

[0188] In certain embodiments, an advertisement may be text (which may be HTML-linked), one or more images (which may be HTML-linked), one or more video, audio, one or more Adobe Flash files, a suitable combination of these, or any other suitable advertisement presented in any suitable digital format on one or more web pages, in one or more emails, or in conjunction with search results requested by a user. Additionally or alternatively, an advertisement may be one or more sponsored stories (e.g., news feeds or offer items on networked system 1102). Sponsored stories may be social actions of users promoted by advertisers (e.g., by displaying social actions in a predetermined area of ​​a user's or other page's profile page, displaying additional information associated with the advertiser, highlighting or otherwise showcasing them in other users' news feeds or offers, or otherwise promoting them). These actions may include "liking" a page, "liking" or commenting on posts on a page, responding to events associated with a page, voting on questions posted on a page, checking in at a location, using an application or playing a game, or "liking" or sharing a website. Advertisers may pay to have social actions promoted. As an example, rather than a restriction, advertisements can be included in the search results on a search results page, where sponsored content is promoted over non-sponsored content.

[0189] In certain embodiments, advertisements may be requested for display on social networking system web pages, third-party web pages, or other pages. Advertisements may appear in dedicated sections of a page, such as in a banner ad area at the top of the page, in a column on the side of the page, in the page's graphical user interface (GUI), in pop-ups, in drop-down menus, in input fields, at the top of page content, or elsewhere relative to the page. Additionally or alternatively, advertisements may be displayed within an application. Advertisements may be displayed within a dedicated page, thus prompting the user to interact with or view the advertisement before the user accesses the page or utilizes the application. Users may view advertisements, for example, through a web browser.

[0190] Users can interact with advertisements in any suitable way. Users can click or otherwise select an advertisement. By selecting an advertisement, users can be directed to a page associated with that advertisement (or the browser or other application being used by the user). On the page associated with the advertisement, users can perform additional actions, such as purchasing the product or service associated with the advertisement, receiving information associated with the advertisement, or subscribing to a newsletter associated with the advertisement. Advertisements with audio or video can be played by selecting components of the advertisement (such as a "play button"). Alternatively, the networked system 1102 can perform or modify specific user actions by selecting an advertisement.

[0191] The advertisement may also include social networking system functionality that users can interact with. As an example, and not by limitation, the advertisement may allow users to "like" or otherwise endorse the advertisement by selecting an icon or link associated with endorsement. As another example, and not by limitation, the advertisement may allow users to search (e.g., by performing a query) for content related to the advertiser. Similarly, users may share the advertisement with another user (e.g., via networked system 1102) or respond to events associated with the advertisement (RSVP) (e.g., via networked system 1102). Additionally or alternatively, the advertisement may include a social networking system environment tailored to the user. As an example, and not by limitation, the advertisement may display information within networked system 1102 about the user's friends who have taken actions related to the topic of the advertisement.

[0192] In a particular embodiment, networking system 1102 can determine the social graph affinity (which may be referred to herein as "affinity") of various social graph entities to each other. Affinity can represent the strength of the relationship or degree of interest between specific objects associated with an online social network (e.g., users, concepts, content, actions, advertisements, other objects associated with the online social network, or any suitable combination thereof). Affinity with objects associated with third-party system 1108 or other suitable systems can also be determined. A total affinity can be established for each user, each topic, or each type of content social graph entity. The total affinity can change based on continued monitoring of actions or relationships associated with social graph entities. Although this disclosure describes determining a particular affinity in a particular manner, this disclosure contemplates determining any suitable affinity in any suitable manner.

[0193] In a particular embodiment, the networking system 1102 may use a proximity coefficient (which may be referred to herein as a "coefficient") to measure or quantify the proximity of a social graph. A coefficient may represent or quantify the strength of a relationship between specific objects associated with an online social network. A coefficient may also represent a probability or function that measures the predicted probability of a user performing a specific action based on the user's interest in that action. In this way, a user's future actions can be predicted based on the user's previous actions, wherein the coefficient may be calculated at least in part based on the history of the user's actions. The coefficient can be used to predict any number of actions, which may be within or outside the online social network. As an example and not by limitation, these actions may include various types of communication (e.g., sending messages, posting content, or commenting on content), various types of observation actions (e.g., accessing or viewing profile pages, media, or other suitable content), various types of consistency information about two or more social graph entities (e.g., information about being in the same group, information about being tagged in the same photo, information about checking in at the same location, or information about regularly attending the same event), or other suitable actions. While this disclosure describes measuring proximity in a particular manner, this disclosure contemplates measuring proximity in any suitable manner.

[0194] In a particular embodiment, the networking system 1102 may use various factors to calculate coefficients. These factors may include, for example, user actions, relationship types between objects, location information, other suitable factors, or any combination thereof. In a particular embodiment, different factors may be weighted differently when calculating coefficients. The weight of each factor may be static, or these weights may change, for example, based on the user, relationship type, action type, user location, etc. The levels of these factors may be combined according to their weights to determine the user's total coefficient. As an example, and not by limitation, both a level and a weight may be assigned to a specific user action, while a level and a related weight may be assigned to the relationship associated with that specific user action (e.g., such that the total weights are 100%). To calculate the coefficient for a user with respect to a specific object, the level assigned to the user action may, for example, include 60% of the total coefficient, while the level assigned to the relationship between the user and the object may include 40% of the total coefficient. In a particular embodiment, when determining the weights of various factors used to calculate coefficients, the networking system 1102 may consider various variables, such as time since accessing the information, decay factor, access frequency, relationship to the information or to the object accessed by the information, relationship to social graph entities connected to the object, short-term or long-term averages of user actions, user feedback, other suitable variables, or any combination thereof. As an example and not by limitation, when calculating coefficients, coefficients may include a decay factor that causes the strength of the signal provided by a particular action to decay over time, making more recent actions more relevant. The ranks and weights may be continuously updated based on ongoing tracking of the actions on which the coefficients are based. Any type of process or algorithm may be used to assign, combine, average, etc., the rank of each factor and the weights assigned to those factors. In a particular embodiment, the networking system 1102 may use a machine learning algorithm trained on historical actions and past user responses, or data from the user obtained by exposing various options to the user and measuring the responses, to determine the coefficients. Although this disclosure describes calculating coefficients in a particular manner, this disclosure contemplates calculating coefficients in any suitable manner.

[0195] In a particular embodiment, the networking system 1102 can calculate coefficients based on user actions. The networking system 1102 can monitor such actions on online social networks, on third-party systems 1108, on other suitable systems, or any combination thereof. Any suitable type of user action can be tracked or monitored. Typical user actions include viewing profile pages, creating or posting content, interacting with content, joining groups, listing and confirming attendance events, checking in at locations, liking specific pages, creating pages, and performing other tasks conducive to social interaction. In a particular embodiment, the networking system 1102 can calculate coefficients based on user actions toward a specific type of content. Content can be associated with online social networks, third-party systems 1108, or other suitable systems. Content can include users, profile pages, posts, news stories, headlines, instant messages, chat room conversations, emails, advertisements, images, videos, music, other suitable objects, or any combination thereof. The networking system 1102 can analyze user actions to determine whether one or more of a plurality of actions indicate affinity toward a topic, content, other users, etc. As an example, and not as a limitation, if a user is likely to frequently post content related to "coffee" or its variations, the network system 1102 can determine that the user has a high coefficient relative to the concept of "coffee." A higher weight and / or grade can be assigned to a specific action or type of action than to other actions, and this higher weight and / or grade can affect the total coefficient calculated. As an example, and not as a limitation, in the case where a first user sends an email to a second user, the weight or grade of that action can be higher than the weight or grade in the case where the first user only views the second user's profile page.

[0196] In a particular embodiment, the networking system 1102 can calculate coefficients based on the type of relationship between specific objects. Referring to the social graph 1200, when calculating coefficients, the networking system 1102 can analyze the number and / or type of edges 1206 connecting a specific user node 1202 and a concept node 1204. As an example, and not by limitation, user nodes 1202 connected by spouse-type edges (indicating that two users are married) can be assigned higher coefficients than multiple user nodes 1202 connected by friend-type edges. That is, depending on the weights assigned to actions and relationships for a particular user, it can be determined that the total intimacy of content about a user's spouse is higher than the total intimacy of content about a user's friends. In a particular embodiment, a user's relationship with another object can affect the weights and / or ranks of the user's actions in calculating the coefficient of that object. As an example, and not by limitation, if a user is tagged in a first photo but only likes a second photo, the networking system 1102 can determine that the user's coefficient relative to the first photo is higher than the user's coefficient relative to the second photo, because a relationship with a tagging type of content can be assigned higher weights and / or ranks than a relationship with a liking type of content. In a particular embodiment, the networking system 1102 may calculate the coefficient of the first user based on the relationship between one or more second users and a specific object. That is, the connections and coefficients of other users and objects can influence the coefficient of the first user and the object. As an example, and not by limitation, if the first user is connected to one or more second users or has a high coefficient with one or more second users, and those second users are connected to a specific object or have a high coefficient with the specific object, then the networking system 1102 may determine that the first user should also have a relatively high coefficient with the specific object. In a particular embodiment, the coefficient may be based on the degree of separation between specific objects. A lower coefficient may indicate a reduced likelihood that the first user shares content objects with users indirectly connected to the first user in the social graph 1200. As an example, and not by limitation, social graph entities that are closer in the social graph 1200 (i.e., with less separation) may have a higher coefficient compared to social graph entities that are more separated in the social graph 1200.

[0197] In a particular embodiment, the networking system 1102 can calculate coefficients based on location information. Objects that are geographically closer to each other can be considered more relevant or of more interest than objects that are farther away. In a particular embodiment, a user's coefficient for a particular object can be based on the proximity of the object's location to the user's current location (or the location of the user's client device 1106). A first user may be more interested in other users or concepts that are closer to the first user. As an example, and not by limitation, if a user is one mile from an airport and two miles from a gas station, the networking system 1102 can determine that the user has a higher coefficient for the airport than for the gas station based on the proximity of the airport to the user.

[0198] In a particular embodiment, the networking system 1102 can perform specific actions about a user based on coefficient information. Coefficients can be used to predict whether a user will perform a specific action based on the user's interest in that action. Coefficients can be used when generating or presenting any type of object to the user, such as advertisements, search results, news stories, media, messages, notifications, or other suitable objects. Where appropriate, coefficients can also be used to rank and order these objects. In this way, the networking system 1102 can provide information relevant to the user's interests and current environment, thereby increasing the likelihood that they will find information of interest. In a particular embodiment, the networking system 1102 can generate content based on coefficient information. Content objects can be provided or selected based on user-specific coefficients. As an example, and not by limitation, coefficients can be used to generate media for a user, where media in which the user has a high total coefficient relative to the media object can be presented to the user. As another example, and not by limitation, coefficients can be used to generate advertisements for a user, where advertisements in which the user has a high total coefficient relative to the advertised object can be presented to the user. In a particular embodiment, the networking system 1102 can generate search results based on coefficient information. Search results for a specific user can be scored or ranked based on a coefficient associated with the search results for that user. As an example, and not as a limitation, on a search results page, a search result corresponding to an object with a higher coefficient may rank higher than a result corresponding to an object with a lower coefficient.

[0199] In a particular embodiment, the networking system 1102 can calculate coefficients in response to a request for coefficients from a particular system or process. Any process can request calculated coefficients for a user to predict possible actions (or the subject of such actions) that a user might take in a given situation. The request may also include a set of weights for various factors used to calculate the coefficients. The request may originate from a process running on an online social network, from a third-party system 1108 (e.g., via an API or other communication channel), or from another suitable system. In response to the request, the networking system 1102 can calculate the coefficients (or, if the coefficient information has been previously calculated and stored, access the coefficient information). In a particular embodiment, the networking system 1102 can measure intimacy with respect to a particular process. Different processes (both internal and external to the online social network) can request coefficients for a specific object or a set of objects. The networking system 1102 can provide a measurement of intimacy related to the specific process requesting the intimacy measurement. In this way, each process receives an intimacy measurement tailored to a different context in which it will use the measurement.

[0200] Combining social graph intimacy and intimacy coefficients, certain implementations may utilize one or more systems, components, elements, functions, methods, actions, or steps disclosed in U.S. Patent Application No. 11 / 503093, filed August 11, 2006; U.S. Patent Application No. 12 / 9711027, filed December 22, 2010; U.S. Patent Application No. 12 / 978265, filed December 23, 2010; and U.S. Patent Application No. 13 / 632869, filed October 1, 2012, each of which is incorporated herein by reference.

[0201] In a particular embodiment, one or more content objects among a plurality of content objects in an online social network may be associated with privacy settings. Privacy settings (or “access settings”) for an object can be stored in any suitable manner, such as associating it with the object, indexing it in an authorization server, using another suitable manner, or any combination thereof. An object’s privacy settings may specify how the online social network can access (e.g., view or share) the object (or specific information associated with the object). Where an object’s privacy settings allow a specific user to access the object, the object can be described as “visible” relative to that user. As an example, rather than by means of restriction, a user of an online social network may specify privacy settings for their profile page that identify a group of users who can access work experience information on the profile page, thereby excluding other users from accessing that information. In a particular embodiment, privacy settings may specify a “blocked list” of users who should not be allowed to access certain information associated with an object. That is, the blocked list may specify one or more users or entities to whom the object is not visible. As an example, rather than by means of restriction, a user may specify a group of users who can not access a photo album associated with them, thereby excluding those users from accessing the album (while still allowing some users not in that group to access the album). In certain embodiments, privacy settings may be associated with specific social graph elements. Privacy settings for a social graph element (e.g., a node or edge) may specify how online social networks can be used to access the social graph element, information associated with the social graph element, or content objects associated with the social graph element. As an example, and not by limitation, a specific concept node 1204 corresponding to a specific photo may have privacy settings specifying that the photo can only be accessed by the user tagged in the photo and their friends. In certain embodiments, privacy settings may allow users to choose to allow or disallow their actions to be recorded by the networked system 1102 or shared with other systems (e.g., third-party system 1108). In certain embodiments, privacy settings associated with an object may specify any suitable granularity for allowing or denying access. As an example, rather than through restrictions, access or denial can be specified for specific users (e.g., only me, my roommate, and my boss), users within a specific separation (e.g., friends, or friends of friends), user groups (e.g., gaming clubs, my family), user networks (e.g., employees of a specific employer, students or alumni of a specific university), all users (“public”), no users (“private”), users of third-party systems 1108, specific applications (e.g., third-party applications, external websites), other suitable users or entities, or any combination thereof. While this disclosure describes the use of specific privacy settings in a particular manner, this disclosure contemplates the use of any suitable privacy settings in any suitable manner.

[0202] In a particular embodiment, one or more servers may be authorization / privacy servers for performing privacy settings. In response to a request from a user (or other entity) for a specific object stored in a data repository, networking system 1102 may send a request for the object to the data repository. If the authorization server determines, based on the privacy settings associated with the object, that the user has been granted permission to access the object, the request can identify the user associated with the request and can be sent only to that user (or that user's client device 1106). If the requesting user has not been granted permission to access the object, the authorization server may prevent the retrieval of the requested object from the data repository or may prevent the sending of the requested object to the user. In a search query scenario, if the querying user has been granted permission to access the object, only that object can be generated as a search result. That is, the object must have visibility visible to the querying user. If the object has visibility not visible to the user, the object can be excluded from the search results. Although this disclosure describes performing privacy settings in a particular manner, this disclosure contemplates performing privacy settings in any suitable manner.

[0203] The foregoing specification has been described with reference to specific exemplary embodiments described herein. Various embodiments and aspects of this disclosure have been described with reference to the details set forth herein, and various embodiments are illustrated in the accompanying drawings. The above description and drawings are illustrative and should not be construed as limiting. Numerous specific details have been described to provide a thorough understanding of various embodiments.

[0204] Additional or alternative embodiments may be embodied in other specific forms without departing from their spirit or essential characteristics. The described embodiments should be considered in all respects as illustrative rather than restrictive. Therefore, the scope of the invention is defined by the appended claims rather than the foregoing description. All variations within the meaning and scope of equivalents of the claims are included within the scope of the claims.

Claims

1. A computer-implemented method, comprising: A client device conducts a video call with a receiving participant device by receiving video data from the receiving participant device via a video data channel established specifically for the video call. The first video captured by the client device is displayed in the digital video call interface; By rendering the video data received via the video data channel, a second video from the receiving participant's device is displayed within the digital video call interface; Augmented reality data indicating augmented reality elements is received from the receiving participant's device via an augmented reality data channel established for the video call with the receiving participant's device; Based on the received augmented reality data, the augmented reality elements are rendered within the first video displayed in the digital video call interface; The method further includes: By rendering the video data received via the video data channel, a second video is displayed within the digital video call interface, depicting a scene from the receiving participant's device that shows additional augmented reality elements exiting the second video. Receive additional augmented reality data of the additional augmented reality element from the receiving participant's device via the augmented reality data channel; and Based on the received augmented reality data, the augmented reality elements are rendered within the first video to depict the scene of the augmented reality elements entering the first video.

2. The computer-implemented method according to claim 1 further includes: Augmented reality elements are rendered within the first video displayed in the digital video call interface, used to share the augmented reality environment scene depicted in the second video from the receiving participant's device, wherein the augmented reality elements modify at least one of the background, foreground, or video filter of the first video.

3. The computer-implemented method according to claim 1, further comprising: Additional augmented reality data indicating modifications to the augmented reality elements is received from the receiving participant's device via the augmented reality data channel; as well as Based on the additional augmented reality data, the augmented reality elements in the first video are modified.

4. The computer-implemented method according to claim 1, further comprising: Identify participant identifiers from the additional augmented reality data; as well as The additional augmented reality element is determined by matching the participant identifier with the client device.

5. The computer-implemented method according to claim 1, further comprising: Identify object vectors from the additional augmented reality data; as well as Using the object vector, the augmented reality element is rendered at a specific location within the first video and the augmented reality element is moved along a specific direction.

6. The computer-implemented method according to claim 1, further comprising: Render augmented reality elements within the first video to display augmented reality effects of user interaction depicted within the first video; as well as A second video, depicting interactions with the recipient user portrayed in the second video, is displayed within the digital video call interface from the recipient participant's device.

7. A non-transitory computer-readable medium storing a plurality of instructions, which, when executed by at least one processor, cause a computing device to: The computing device conducts a video call with the receiving participant device by receiving video data from the receiving participant device via a video data channel established for the video call. The first video captured by the computing device is displayed within the digital video call interface; By rendering the video data received via the video data channel, a second video from the receiving participant's device is displayed within the digital video call interface; Augmented reality data indicating augmented reality elements is received from the receiving participant's device via an augmented reality data channel established for the video call with the receiving participant's device; Based on the received augmented reality data, the augmented reality elements are rendered within the first video displayed in the digital video call interface; By rendering the video data received via the video data channel, a second video is displayed within the digital video call interface, depicting a scene from the receiving participant's device that shows additional augmented reality elements exiting the second video. Additional augmented reality data of the additional augmented reality element is received from the receiving participant's device via the augmented reality data channel. as well as Based on the received augmented reality data, the augmented reality elements are rendered within the first video to depict the scene of the augmented reality elements entering the first video.

8. The non-transitory computer-readable medium of claim 7, further comprising a plurality of instructions which, when executed by the at least one processor, cause the computing device to perform the following operations: Detecting user movement interactions within the first video captured by the computing device; and Based on the user's mobile interaction, the augmented reality elements in the first video are modified.

9. The non-transitory computer-readable medium of claim 7, further comprising a plurality of instructions which, when executed by the at least one processor, cause the computing device to perform the following operations: Render the augmented reality elements within the first video to display augmented reality effects of user interaction depicted within the first video; and A second video, depicting interactions with the recipient user portrayed in the second video, is displayed within the digital video call interface, originating from the recipient participant's device.

10. The non-transitory computer-readable medium of claim 9, further comprising a plurality of instructions which, when executed by the at least one processor, cause the computing device to perform the following operations: Identify the choice in the augmented reality data that directs the computing device to the primary participant; and Based on the selection that designates the computing device as the primary participant, the augmented reality effect in the first video on the computing device is rendered differently from the additional augmented reality effect displayed in the second video.

11. The non-transitory computer-readable medium of claim 9, further comprising, when executed by the at least one processor, a plurality of instructions causing the computing device to perform the following operations: receiving augmented reality data indicating the augmented reality elements by filtering a set of augmented reality data in the augmented reality data channel.

12. A shared augmented reality video call system, comprising: At least one server, configured to establish a shared augmented reality video call between the first participant device and the second participant device by establishing a video data channel and an augmented reality data channel between the first participant device and the second participant device, wherein: The first participant device acquires a first video stream and sends the first video stream to the second participant device via the video data channel; The second participant device acquires a second video stream and sends the second video stream to the first participant device via the video data channel; The first participant device sends an augmented reality identifier to the second participant device via the augmented reality data channel; The second participant device renders the second video stream using augmented reality elements associated with the augmented reality identifier; and in: The first participant device transmits, via the video data channel, a first video stream depicting the augmented reality elements within the first video stream to the second participant device; and Once the augmented reality identifier is sent to the second participant device via the augmented reality data channel: The first participant device transmits a first video stream, depicting the augmented reality element exiting the first video stream, to the second participant device via the video data channel; and The second participant device renders the second video stream with the augmented reality elements of the scene entering the second video stream, which are associated with the augmented reality identifier.

13. The shared augmented reality video call system according to claim 12, wherein, The first participant device renders the first video stream using additional augmented reality elements associated with the augmented reality identifier to share the augmented reality environment scene depicted within the second video stream from the second participant device.

14. The shared augmented reality video call system according to claim 12, wherein: The first participant device sends an additional augmented reality identifier to indicate interaction with the augmented reality element; as well as The second participant device renders the second video stream using modifications to the augmented reality elements based on the additional augmented reality identifier.

15. The shared augmented reality video call system according to claim 12, wherein: The first participant device sends the augmented reality identifier, which indicates the object vector of the augmented reality element, to the second participant device via the augmented reality data channel; and The second participant device uses the object vector to render the second video stream using the augmented reality element associated with the augmented reality identifier, which is located at a specific position and moves in a specific direction.

16. The shared augmented reality video call system according to claim 12, wherein: The second participant device renders the second video stream using augmented reality elements for displaying augmented reality effects of user interactions depicted within the second video stream; as well as The first participant's device renders the first video stream using additional augmented reality effects depicting user interactions within the first video stream.

17. The shared augmented reality video call system according to claim 12, wherein: The first participant device sends augmented reality identifiers to multiple participant devices via the augmented reality data channel; as well as The multiple participant devices use augmented reality elements associated with the augmented reality identifier to render the video stream.

Images