Enhanced dual video call with augmented reality streaming

Abstract

Embodiments include a method performed by a processor of a receiving wireless device for rendering augmented reality content. The method may include receiving real image data from a transmitting wireless device via a first data stream, and receiving augmented reality (AR) content from the transmitting wireless device via a second data stream as an alternative video stream, where the AR content is for display with the real image data. The method may also include displaying the AR content in conjunction with the real image data. In some embodiments, the first data stream and the second data stream are real-time transport protocol (RTP) streams.

Description

[0001] Related Applications

[0002] This application claims the benefit of priority to U.S. non-provisional application No. 17 / 818,989, filed on August 11, 2022; the entire contents of which are incorporated herein by reference. Background Art

[0003] Augmented reality (AR) is an interactive experience of a real-world environment in which objects residing in the real world are enhanced by computer-generated sensory information. There are a variety of use cases where this computer-generated sensory information is received through a server. For example, when the computer-generated sensory information is received from the server, the client can use a synthesizer and augment the actual real-world image with the computer-generated sensory image in real time. Thus, the computer-generated sensory image can be overlapped with the real-world image to provide additional information to the user. Such applications have been implemented in the fields of mapping, tourism, education, maintenance and repair, medical, design, modeling, and retail applications. Summary of the invention

[0004] Various aspects include a method for rendering augmented reality content performed by a processor of a receiving wireless device. Various aspects of the method may include: receiving real image data from a sending wireless device via a first data stream; receiving augmented reality (AR) content from the sending wireless device via a second data stream as an alternative video stream, wherein the AR content is for display with the real image data, and displaying the AR content in conjunction with the real image data.

[0005] In some aspects, the first data stream and the second data stream may be real-time transport protocol (RTP) streams, and the AR content may be received as part of an RTP extension header. In some aspects, the first data stream and the second data stream may be RTP control protocol (RTCP) streams, and the AR content may be received as part of an RTCP application definition (APP) packet.

[0006] Some aspects may also include: receiving a call setup request from a sending wireless device; configuring a modem of a receiving wireless device based on the call setup request to receive data via a first data stream and a second data stream, and establishing a dual video call including the first data stream and the second data stream with the sending wireless device according to RFC4796.

[0007] In some aspects, receiving real image data from a transmitting wireless device via a first data stream may further include receiving encoded real image data from the transmitting wireless device via the first data stream, and receiving AR content from the transmitting wireless device via a second data stream may further include receiving encoded AR content from the transmitting wireless device via the second data stream. Some aspects may further include decoding the encoded real image data to obtain decoded real image data, and decoding the encoded AR content to obtain decoded AR content. In some aspects, displaying AR content in conjunction with real image data may further include displaying decoded AR content in conjunction with decoded real image data.

[0008] Some aspects may also include correlating the first data stream and the second data stream with an RTP timestamp according to RFC 3550, wherein displaying the AR content in conjunction with the real image data may further include displaying the AR content in conjunction with the real image data based on the RTP timestamp.

[0009] In some aspects, displaying AR content in conjunction with real image data may also include: sending real image data and coordinate transformations to a monitor communicatively coupled to a receiving wireless device so that the monitor displays the real image data and coordinate transformations, and sending AR content and coordinate transformations to an AR user equipment communicatively coupled to the receiving wireless device so that the AR user equipment displays the AR content and coordinate transformations.

[0010] Some aspects may also include synthesizing the AR content with real image data to form a synthetic two-dimensional (2D) image, wherein displaying the AR content in combination with the real image data may also include sending the synthetic 2D image to an AR user equipment communicatively coupled to a receiving wireless device so that the AR user equipment displays the synthetic 2D image.

[0011] Some aspects may also include synthesizing the AR content with the real image data to form a synthesized 2D image, wherein displaying the AR content in conjunction with the real image data may further include displaying the synthesized 2D image on a display terminal of the receiving wireless device.

[0012] In some aspects, receiving AR content from a transmitting wireless device via a second data stream may further include receiving a synthetic 2D image synthesized by the AR content and real image data from the transmitting wireless device via the second data stream, wherein displaying the AR content in combination with the real image data may further include displaying the synthetic 2D image.

[0013] Other aspects include a computing device having a processor configured to perform one or more operations of any of the methods outlined above. Other aspects include a computing device having components for performing the functions of any of the methods outlined above. Other aspects include a non-transitory processor-readable storage medium having stored thereon processor-executable instructions configured to cause a processor of the computing device to perform the operations of any of the methods outlined above. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the claims and, together with the general description and detailed description given, serve to explain the features herein.

[0015] Figure 1A is a system block diagram illustrating an example communication system suitable for implementing any of the various embodiments.

[0016] Figure 1B is a system block diagram illustrating an example decomposed base station architecture for a wireless communication system suitable for implementing any of the various embodiments.

[0017] Figure 2 is a component block diagram illustrating an example computing and wireless modem system suitable for implementing any of the various embodiments.

[0018] Figure 3 is a component block diagram illustrating a software architecture including a radio protocol stack for user plane and control plane in wireless communications suitable for implementing any of the various embodiments.

[0019] Figure 4 is a component block diagram illustrating an example system configured to establish secure device-to-device communications between two wireless devices according to some embodiments.

[0020] Figure 5 is a message flow diagram illustrating message exchanges between a sending wireless device, a network, and a receiving wireless device during call setup and an active call according to various embodiments.

[0021] Figure 6 is a system block diagram of an example AR communication system that illustrates an alternative (ALT) stream as an augmented reality (AR) stream according to some embodiments.

[0022] Figure 7 is a system block diagram illustrating an example AR communication system that implements ALT streams as AR streams according to some embodiments.

[0023] Figure 8 is a system block diagram illustrating an example AR communication system that implements ALT streams as AR streams according to some embodiments.

[0024] Fig. 9 is a system block diagram illustrating an example AR communication system that implements an ALT stream as an AR composite image stream according to some embodiments.

[0025] Fig. 10A is a process flow diagram of an example method 1000a for rendering AR content according to various embodiments.

[0026] FIG. 10B to FIG. 10E is a process flow diagram of example operations 1000b - 1000e that may be performed as part of method 1000a for rendering AR content, according to some embodiments.

[0027] Fig.11 is a component block diagram of a network computing device suitable for use with various embodiments.

[0028] Fig.12 is a component block diagram of a wireless device suitable for use with various embodiments. DETAILED DESCRIPTION

[0029] Various embodiments will be described in detail with reference to the accompanying drawings. Where possible, the same reference numerals will be used throughout the drawings to refer to the same or similar parts. References to specific examples and implementations are for illustrative purposes and are not intended to limit the scope of the claims.

[0030] Various embodiments include methods and processors of wireless devices that receive augmented reality (AR) information from a sending wireless device via an ALT stream or an auxiliary stream to render and display AR content in combination with real image data. Various embodiments may include a processor of a receiving wireless device (e.g., a mobile phone) configured to receive real image data (i.e., data representing a real-world image captured by a camera / sensor) from a sending wireless device (e.g., a mobile phone) via a first data stream. The processor of the receiving wireless device may be further configured to receive AR content from the sending wireless device via a second data stream, wherein the second data stream is an ALT video stream.

[0031] The term "system on chip" (SoC) is used herein to refer to a single integrated circuit (IC) chip that includes multiple resources and / or processors integrated on a single substrate. A single SoC may include circuits for digital, analog, mixed signal, and radio frequency functions. A single SoC may also include any number of general and / or specialized processors (digital signal processors, modem processors, video processors, etc.), storage blocks (e.g., ROM, RAM, flash memory, etc.), and resources (e.g., timers, voltage regulators, oscillators, etc.). The SoC may also include software for controlling the integrated resources and processors and for controlling peripheral devices.

[0032] The term "system-in-package" (SIP) may be used herein to refer to a single module or package that contains multiple resources, computing units, cores and / or processors on two or more IC chips, substrates, or SoCs. For example, a SIP may include a single substrate on which multiple IC chips or semiconductor dies are stacked in a vertical configuration. Similarly, a SIP may include one or more multi-chip modules (MCMs) on which multiple ICs or semiconductor dies are packaged into a unified substrate. A SIP may also include multiple independent SoCs that are coupled together via high-speed communication circuits and packaged in close proximity, such as on a single motherboard or in a single wireless device. The proximity of the SoCs facilitates high-speed communication and sharing of memory and resources.

[0033] As used herein, the terms "network", "system", "wireless network", "cellular network" and "wireless communication network" may interchangeably refer to a portion or all of a wireless network of a carrier associated with a wireless device and / or a subscription on a wireless device. The techniques described herein may be used in various wireless communication networks, such as code division multiple access (CDMA), time division multiple access (TDMA), FDMA, orthogonal FDMA (OFDMA), single carrier FDMA (SC-FDMA), and other networks. In general, any number of wireless networks may be deployed in a given geographic area. Each wireless network may support at least one radio access technology, which may operate on one or more frequencies or frequency ranges. For example, a CDMA network may implement Universal Terrestrial Radio Access (UTRA) (including Wideband Code Division Multiple Access (WCDMA) standards), CDMA2000 (including IS-2000, IS-95 and / or IS-856 standards), etc. In another example, a TDMA network may implement GSM Enhanced Data Rates for GSM Evolution (EDGE). For example, OFDMA networks can implement Evolved UTRA (E-UTRA), IEEE 802.11 (WiFi), IEEE 802.16 (WiMAX), IEEE802.20, Flash- Etc. Reference may be made to wireless networks using the Long Term Evolution (LTE) standard, and thus the terms "Evolved Universal Terrestrial Radio Access," "E-UTRAN," and "Evolved Node B" may also be used interchangeably herein to refer to wireless networks. However, such references are provided merely as examples and are not intended to exclude wireless networks using other communication standards. For example, although various third generation (3G) systems, fourth generation (4G) systems, and fifth generation (5G) systems are discussed herein, those systems are cited merely as examples and may be replaced with future generation systems (e.g., sixth generation (6G) or higher generation systems) in various embodiments.

[0034] As used herein, the terms "real image data" and "real-world image data" may interchangeably refer to data representing a captured two-dimensional (2D) image. The real-world image data may correspond to a real-world perspective captured by a camera or other optical sensor. The real-world image data may be used by a processor to render and display a real-world image.

[0035] As used herein, the term "augmented reality (AR) content" may refer to content generated from real-world image data. AR content may include any number and type of images, characters, symbols, and visual representations that may be overlaid on real-world images to provide additional information to the user.

[0036] As used herein, the terms "first data stream", "main data stream" and "MAIN data stream" interchangeably refer to a data stream established between two wireless devices capable of communicating media data. The MAIN data stream may be a "MAIN stream" as described in Request for Comments (RFC) 4796.

[0037] As used herein, the terms "secondary data stream", "auxiliary data stream" and "ALT data stream" interchangeably refer to a data stream between two wireless devices capable of communicating media data. The ALT data stream may be an "ALT stream" as described with reference to RFC 4796. The ALT data stream is a data stream that is an alternative data stream to the MAIN data stream.

[0038] For a conventional multimedia call, a calling wireless device may establish a communication link with a called wireless device, wherein multiple media streams are enabled for a single call session (e.g., a Session Initiation Protocol ("SIP") session). Typically, in such a basic configuration, there is one active video stream and one corresponding active audio stream. For example, a user of a wireless device may initiate a video call, and a communication session may be established that includes a video stream showing either a front camera view or a rear camera view and an accompanying audio stream. In a dual video application, more than one video stream may be enabled. For example, a user may operate a wireless device to initiate a call session with another wireless device, wherein the resulting connection enables streaming of a front camera view in a first video stream and a rear camera view in a second video stream.

[0039] In a dual video application, one video stream is selected as the main stream or MAIN stream and the second video stream is selected as the auxiliary stream or ALT stream. RFC 4796 standardizes the process negotiation between the MAIN stream and the ALT stream in a dual video application. As in the aforementioned example of implementing dual video streams, one video stream (e.g., the front camera view) may be selected as the MAIN stream and the other video stream (e.g., the rear camera view) may be selected as the ALT stream. The MAIN stream and the ALT stream may be established as a real-time transport protocol (RTP) stream and / or may be configured as an RTP control protocol (RTCP) stream.

[0040] Various embodiments improve the dual video functionality as standardized in RFC 4796 by configuring the ALT stream as an AR stream, wherein AR content or data is transmitted from one wireless device to another wireless device in order to render the AR content in conjunction with the corresponding real-world image transmitted via the MAIN stream. Since the MAIN stream can be conventionally configured as a video stream and the ALT stream can be configured as an AR stream, wherein both the MAIN stream and the ALT stream are configured as RTP streams sent from the same source (i.e., the sending wireless device), timing information (e.g., RTP timestamps) can be maintained at the receiving end (i.e., the receiving wireless device) to "stitch" the dual streams together to display the AR content in conjunction with the real-world image relative to real-time. Therefore, configuring the ALT stream as an AR stream can enhance the video call experience by providing additional functionality in addition to audio and video streaming only. Similar streaming concepts can be applied to dual video streaming of other extended reality ("XR") environments, including mixed reality ("MR") and virtual reality ("VR").

[0041] Various embodiments allow the ALT stream to be configured as an AR stream capable of conveying AR content in conjunction with real-world image data. Various embodiments may implement a processor of a receiving wireless device for rendering augmented reality content, wherein the processor is configured to receive real image data from a sending wireless device via a first data stream (i.e., a MAIN stream), receive AR content from the sending wireless device via a second data stream that is an ALT video stream (i.e., where the AR content is used to be displayed together with the real image data), and further configured to display the AR content in conjunction with the real image data.

[0042] In some embodiments, the first data stream and the second data stream may be RTP streams, and the AR content may be received by a processor of the receiving wireless device as part of an RTP extension header. In some embodiments, the first data stream and the second data stream may be Real-time Transport Protocol (RTP) Control Protocol (RTCP) streams, and the AR content may be received by a processor of the receiving wireless device as part of an RTCP custom payload (e.g., an RTCP Application Definition (APP) packet as described in RFC3550). In some embodiments, the processor of the receiving wireless device may associate the first data stream and the second data stream with an RTP timestamp according to RFC 3550, wherein the AR content may be displayed based on the RTP timestamp in combination with the real image data.

[0043] FIG. 1 is a system block diagram illustrating an example communication system 100 suitable for implementing any of the various embodiments. The communication system 100 may be a 5G New Radio (NR) network, or any other suitable network such as an LTE network, a 5G network, etc. Although FIG. 1 illustrates a 5G network, later generation networks may also include the same or similar elements. Therefore, references to 5G networks and 5G network elements in the following description are for illustrative purposes and are not intended to be limiting.

[0044] The communication system 100 may include a heterogeneous network architecture including a core network 140 and various mobile devices (illustrated as wireless devices 120a-120e in FIG. 1 ). The communication system 100 may also include several base stations (illustrated as BS110a, BS110b, BS110c, and BS110d) and other network entities. A base station is an entity that communicates with a wireless device and may also be referred to as a node B, an LTE evolved node B (eNodeB or eNB), an access point (AP), a radio head, a transmit receive point (TRP), a new radio base station (NR BS), a 5G NodeB (NB), a next generation NodeB (gNodeB or gNB), and the like. Each base station may provide communication coverage for a specific geographic area. In 3GPP, the term "cell" may refer to a coverage area of ​​a base station, or a base station subsystem serving the coverage area, or a combination thereof, depending on the context in which the term is used. The core network 140 may be any type of core network, such as an LTE core network (e.g., an evolved packet core (EPC) network), a 5G core network, etc.

[0045] Base stations 110a-110d may provide communication coverage for macro cells, pico cells, femto cells, another type of cell, or a combination thereof. A macro cell may cover a relatively large geographic area (e.g., a radius of several thousand meters) and may allow unrestricted access by mobile devices with service subscriptions. A pico cell may cover a relatively small geographic area and may allow unrestricted access by mobile devices with service subscriptions. A femto cell may cover a relatively small geographic area (e.g., a residence) and may allow restricted access by mobile devices associated with the femto cell (e.g., mobile devices in a closed subscriber group (CSG)). A base station for a macro cell may be referred to as a macro BS. A base station for a pico cell may be referred to as a pico BS. A base station for a femto cell may be referred to as a femto BS or a home BS. In the example illustrated in FIG. 1 , base station 110a may be a macro BS for a macro cell 102a, base station 110b may be a pico BS for a pico cell 102b, and base station 110c may be a femto BS for a femto cell 102c. Base stations 110a-110d may support one or more (e.g., three) cells. The terms “eNB,” “base station,” “NR BS,” “gNB,” “TRP,” “AP,” “Node B,” “5G NB,” and “cell” may be used interchangeably herein.

[0046] In some examples, the cell may not be stationary, and the geographic area of ​​the cell may move depending on the location of the mobile base station. In some examples, the base stations 110a-110d may be interconnected to each other and to one or more other base stations or network nodes (not shown) in the communication system 100 via various types of backhaul interfaces (such as direct physical connections, virtual networks, or combinations thereof) using any suitable transport network.

[0047] The base stations 110a-110d may communicate with the core network 140 over a wired or wireless communication link 126. The wireless devices 120a-120e may communicate with the base stations 110a-110d over a wireless communication link 122.

[0048] The wired communication link 126 may use various wired networks (e.g., Ethernet, TV cable, telephone, fiber optic, and other forms of physical network connections) that may use one or more wired communication protocols (such as Ethernet, Point-to-Point Protocol, High-Level Data Link Control (HDLC), Advanced Data Communications Control Protocol (ADCCP), and Transmission Control Protocol / Internet Protocol (TCP / IP)).

[0049] The communication system 100 may also include a relay station (e.g., relay BS 110d). A relay station is an entity that can receive data from an upstream station (e.g., a base station or a mobile device) and send the data to a downstream station (e.g., a wireless device or a base station). A relay station may also be a mobile device that can relay transmissions for other wireless devices. In the example illustrated in FIG. 1 , a relay station 110d may communicate with a macro base station 110a and a wireless device 120d to facilitate communication between the base station 110a and the wireless device 120d. A relay station may also be referred to as a relay base station, a relay base station, a relay, etc.

[0050] The communication system 100 may be a heterogeneous network including different types of base stations (e.g., macro base stations, pico base stations, femto base stations, relay base stations, etc.). These different types of base stations may have different transmit power levels, different coverage areas, and different impacts on interference in the communication system 100. For example, a macro base station may have a high transmit power level (e.g., 5 watts to 40 watts), while a pico base station, a femto base station, and a relay base station may have a lower transmit power level (e.g., 0.1 watt to 2 watts).

[0051] A network controller 130 may be coupled to a set of base stations and may provide coordination and control of these base stations. The network controller 130 may communicate with the base stations via backhaul. The base stations may also communicate with each other directly or indirectly, for example, via wireless or wired backhaul.

[0052] The wireless devices 120a, 120b, 120c may be dispersed throughout the communication system 100, and each wireless device may be stationary or mobile. A wireless device may also be referred to as an access terminal, terminal, mobile station, subscriber unit, station, user equipment (UE), etc.

[0053] The macro base station 110a may communicate with the communication network 140 over a wired or wireless communication link 126. The wireless devices 120a, 120b, 120c may communicate over a wireless communication link 122 with the base stations 110a-110d.

[0054] The wireless communication links 122, 124 may include multiple carrier signals, frequencies, or frequency bands, each of which may include multiple logical channels. The wireless communication links 122 and 124 may utilize one or more radio access technologies (RATs). Examples of RATs that may be used in the wireless communication links include: 3GPP LTE, 3G, 4G, 5G (e.g., NR), GSM, CDMA, WCDMA, Worldwide Interoperability for Microwave Access (WiMAX), Time Division Multiple Access (TDMA), and other mobile phone communication technology cellular RATs. Further examples of RATs that may be used in one or more of the various wireless communication links 122, 124 within the communication system 100 include medium-range protocols such as Wi-Fi, LTE-U, LTE-Direct, LAA, MuLTEfire, and relatively short-range RATs such as ZigBee, Bluetooth, and Bluetooth Low Energy (LE).

[0055] Certain wireless networks (e.g., LTE) use orthogonal frequency division multiplexing (OFDM) on the downlink and single carrier frequency division multiplexing (SC-FDM) on the uplink. OFDM and SC-FDM divide the system bandwidth into multiple (K) orthogonal subcarriers, which are also often referred to as frequency modulation, frequency bins, etc. Each subcarrier can be modulated with data. In general, modulation symbols are transmitted in the frequency domain using OFDM and conveyed in the time domain using SC-FDM. The spacing between adjacent subcarriers can be fixed, and the total number of subcarriers (K) can depend on the system bandwidth. For example, the spacing of the subcarriers can be 15kHz, and the minimum resource allocation (called a "resource block") can be 12 subcarriers (or 180kHz). Therefore, for a system bandwidth of 1.25 megahertz (MHz), 2.5MHz, 5MHz, 10MHz, or 20MHz, the nominal fast file transfer (FFT) size can be equal to 128, 256, 512, 1024, or 2048, respectively. The system bandwidth can also be divided into multiple subbands. For example, a subband may cover 1.08 MHz (ie, 6 resource blocks), and there may be 1, 2, 4, 8, or 16 subbands for system bandwidths of 1.25 MHz, 2.5 MHz, 5 MHz, 10 MHz, or 20 MHz, respectively.

[0056] Although the description of some embodiments may use terms and examples associated with LTE technology, some embodiments may be applicable to other wireless communication systems (such as new radio (NR) or 5G networks). NR may utilize OFDM with a cyclic prefix (CP) on the uplink (UL) and downlink (DL), and includes support for half-duplex operation using time division duplex (TDD). A single component carrier bandwidth of 100 MHz may be supported. NR resource blocks may span 12 subcarriers with a subcarrier bandwidth of 75 kHz over a duration of 0.1 milliseconds (ms). Each radio frame may include 50 subframes with a length of 10 ms. Thus, each subframe may have a length of 0.2 ms. Each subframe may indicate a link direction (i.e., DL or UL) for data transmission, and the link direction of each subframe may be dynamically switched. Each subframe may include DL / UL data and DL / UL control data. Beamforming may be supported and the beam direction may be dynamically configured. Multiple-input multiple-output (MIMO) transmission with precoding may also be supported. MIMO configurations in the DL may support up to 8 transmit antennas (with multi-layer DL transmission of up to eight streams) and up to two streams per wireless device. Multi-layer transmission of up to 2 streams per wireless device may be supported. Aggregation of multiple cells may be supported using up to eight serving cells. Alternatively, NR may support different air interfaces other than OFDM-based air interfaces.

[0057] Some mobile devices may be considered as machine type communication (MTC) or evolved or enhanced machine type communication (eMTC) mobile devices. MTC and eMTC mobile devices include, for example, robots, remote devices, sensors, meters, monitors, location tags, etc., which can communicate with a base station, another device (e.g., a remote device), or some other entity. For example, a wireless node may provide a connection for or to a network (e.g., a wide area network such as the Internet or a cellular network) via a wired or wireless communication link. Some mobile devices may be considered as Internet of Things (IoT) devices, or may be implemented as NB-IoT (narrowband Internet of Things) devices. Wireless devices 120a-120e may be included inside a housing that houses components of the wireless device, such as a processor component, a memory component, similar components, or a combination thereof.

[0058] In general, any number of communication systems and any number of wireless networks may be deployed in a given geographic area. Each communication system and wireless network may support a specific radio access technology (RAT) and may operate on one or more frequencies. RAT may also be referred to as radio technology, air interface, etc. Frequency may also be referred to as carrier, frequency channel, etc. Each frequency may support a single RAT in a given geographic area to avoid interference between communication systems of different RATs. In some cases, 4G / LTE and / or 5G / NR RAT networks may be deployed. For example, a 5G non-standalone (NSA) network may use 4G / LTE RAT on the 4G / LTE RAN side of a 5G NSA network and simultaneously use 5G / NR RAT on the 5G / NR RAN side of a 5GNSA network. Both 4G / LTE RAN and 5G / NR RAN may be connected to each other and to a 4G / LTE core network (e.g., an evolved packet core (EPC) network) in a 5G NSA network. Other example network configurations may include a 5G standalone (SA) network in which 5G / NR RAN is connected to a 5G core network.

[0059] In some embodiments, two or more wireless devices 120a-120e (e.g., illustrated as wireless device 120a and wireless device 120e) may use one or more side link channels 124 to communicate directly (e.g., without using base station 110a-110d as an intermediary to communicate with each other). For example, the wireless devices 120a-120e may communicate using peer-to-peer (P2P) communication, device-to-device (D2D) communication, vehicle-to-everything (V2X) protocols (which may include vehicle-to-vehicle (V2V) protocols, vehicle-to-infrastructure (V2I) protocols, or similar protocols), mesh networks, or similar networks, or combinations thereof. In this case, the wireless devices 120a-120e may perform scheduling operations, resource selection operations, and other operations described elsewhere herein as being performed by base station 110a.

[0060] The wireless device 120a-120e may be configured to monitor channel conditions and report the channel conditions to the base station 110a-110d. For example, the channel conditions may be indicated in channel state information (CSI) reported by the wireless device 120a-120e to the base station 110a-110d. The CSI reported by the wireless device 120a-120e may include a channel quality indicator (CQI) index value indicated in a channel state feedback (CSF) report sent from the wireless device 120a-120e to the base station 110a-110d. The CSI may be reported by the wireless device 120a-120e to the base station 110a-110d via a physical uplink control channel (PUCCH) or a physical uplink shared channel (PUSCH). The CQI index value may be an observed or estimated channel measurement value sent by the wireless device 120a-120e to the base station 110a-110d as an index value indicating the channel quality. The CQI index value may be an integer value, such as a value of 0-15, which indicates the quality of the DL channel observed or estimated by the wireless device 120a-120e.

[0061] The base stations 110a-110d may be configured to select the wireless devices 120a-120e to receive the broadcast packets based on a channel quality estimate, such as based on the channel conditions reported by the wireless devices 120a-120e in the CSI reported by the wireless devices 120a-120e. The CQI index value may be used by the base stations 110a-110d to determine a modulation and coding scheme (MCS) value for the wireless devices 120a-120e. The base stations 110a-110d may construct an MCS pool that contains the union of all MCS values ​​determined based on CSI reports from the wireless devices 120a-120e in the cell. During rate control operations, the base stations 110a-110d may select a minimum MCS value to cover a percentage of the MCS pool, such as 25%, 50%, 100%, and select the wireless devices 120a-120e having an MCS at or above the minimum MCS value to receive the broadcast packets. Wireless devices 120a-120e having an MCS lower than the minimum MCS value may not be selected to receive the broadcast packet.

[0062] Figure 1B is a system block diagram illustrating an example decomposed base station 160 architecture that may be part of a 5G network suitable for delivering AR content via RTP streaming. Figure 1A and Figure 1B, the decomposed base station 160 architecture may include one or more central units (CU) 162, which may communicate directly with the core network 180 via a backhaul link, or indirectly with the core network 180 through one or more decomposed base station units (such as a near real-time (near RT) RAN intelligent controller (RIC) 164 via an E2 link, or a non-real-time (non-RT) RIC 168 associated with a service management and orchestration (SMO) framework 166, or both). The CU 162 may communicate with one or more distributed units (DU) 170 via corresponding midhaul links (such as an F1 interface). The DU 170 may communicate with one or more radio units (RU) 172 via corresponding fronthaul links. The RU 172 may communicate with corresponding UEs 120 via one or more radio frequency (RF) access links. In some implementations, a user equipment (UE) (such as a vehicle safety system 104) may be served by multiple RUs 172 simultaneously.

[0063] Each of the units (i.e., CU 162, DU 170, RU 172), as well as near-RT RIC 164, non-RT RIC 168, and SMO framework 166 may include one or more interfaces, or be coupled to one or more interfaces configured to receive or send signals, data, or information (collectively referred to as signals) via a wired or wireless transmission medium. Each of the units or an associated processor or controller that provides instructions to the communication interface of these units may be configured to communicate with one or more of the other units via a transmission medium. For example, the unit may include a wired interface that is configured to receive or send signals to one or more of the other units on a wired transmission medium. Additionally, these units may include a wireless interface that may include a receiver, a transmitter, or a transceiver (such as a radio frequency (RF) transceiver) that is configured to receive or send signals, or both, to one or more of the other units on a wireless transmission medium.

[0064] In some aspects, CU 162 may host one or more higher layer control functions. Such control functions may include radio resource control (RRC), packet data convergence protocol (PDCP), service data adaptation protocol (SDAP), etc. Each control function may be implemented with an interface configured to communicate signals with other control functions hosted by CU 162. CU 162 may be configured to handle user plane functionality (i.e., central unit-user plane (CU-UP)), control plane functionality (i.e., central unit-control plane (CU-CP)), or a combination thereof. In some specific implementations, CU 162 may be logically divided into one or more CU-UP units and one or more CU-CP units. When implemented in an O-RAN configuration, the CU-UP unit may communicate bidirectionally with the CU-CP unit via an interface (such as an E1 interface). As needed, CU 162 may be implemented to communicate with DU 170 for network control and signaling.

[0065] DU 170 may correspond to a logical unit that includes one or more base station functions for controlling the operation of one or more RUs 172. In some aspects, DU 170 may host one or more of a radio link control (RLC) layer, a medium access control (MAC) layer, and one or more high physical (PHY) layers (such as modules for forward error correction (FEC) coding and decoding, scrambling, modulation and demodulation, etc.) at least in part according to a functional partitioning such as that defined by the Third Generation Partnership Project (3GPP). In some aspects, DU 170 may further host one or more low PHY layers. Each layer (or module) may be implemented using an interface that is configured to communicate signals with other layers (and modules) hosted by DU 170 or with control functions hosted by CU 162.

[0066] The lower layer functionality may be implemented by one or more RUs 172. In some deployments, the RUs 172 controlled by the DUs 170 may correspond to logical nodes that host RF processing functions or low PHY layer functions (such as performing fast Fourier transforms (FFTs), inverse FFTs (iFFTs), digital beamforming, physical random-access channel (PRACH) extraction and filtering, etc.), or both, based at least in part on functional partitioning (such as lower layer functional partitioning). In such an architecture, the RUs 172 may be implemented to handle over-the-air (OTA) communications with one or more UEs 120. In some specific implementations, real-time and non-real-time aspects of control and user plane communications with the RUs 172 may be controlled by the corresponding DUs 170. In some scenarios, this configuration may enable the DUs 170 and CUs 162 to be implemented in a cloud-based radio access network (RAN) architecture (such as a vRAN architecture).

[0067] The SMO framework 166 may be configured to support RAN deployment and provisioning of non-virtualized network elements and virtualized network elements. For non-virtualized network elements, the SMO framework 166 may be configured to support the deployment of dedicated physical resources for RAN coverage requirements, which may be managed via an operation and maintenance interface (such as an O1 interface). For virtualized network elements, the SMO framework 166 may be configured to interact with a cloud computing platform (such as an open cloud (O-Cloud) 176) to perform network element lifecycle management (such as to instantiate virtualized network elements) via a cloud computing platform interface (such as an O2 interface). Such virtualized network elements may include, but are not limited to, CU 162, DU 170, RU 172, and near-RT RIC 164. In some specific implementations, the SMO framework 166 may communicate with hardware aspects of the 4G RAN (such as an open eNB (O-eNB) 174) via the O1 interface. Additionally, in some specific implementations, the SMO framework 166 may communicate directly with one or more RUs 172 via the O1 interface. The SMO framework 166 may also include a non-RT RIC 168 configured to support the functionality of the SMO framework 166 .

[0068] The non-RT RIC 168 may be configured to include logic functions that enable non-real-time control and optimization of RAN elements and resources, artificial intelligence / machine learning (AI / ML) workflows including model training and updating, or policy-based guidance of applications / features in the near-RT RIC 164. The non-RT RIC 168 may be coupled to or in communication with the near-RT RIC 164 (such as via an A1 interface). The near-RT RIC 164 may be configured to include logic functions that enable near-real-time control and optimization of RAN elements and resources via data collection and actions through an interface (such as via an E2 interface) that connects one or more CUs 162, one or more DUs 170, or both, and the O-eNB with the near-RT RIC 164.

[0069] In some implementations, in order to generate an AI / ML model to be deployed in the near-RT RIC 164, the non-RT RIC 168 may receive parameters or external enrichment information from an external server. Such information may be utilized by the near-RT RIC 164 and may be received from a non-network data source or from a network function at the SMO framework 166 or the non-RT RIC 168. In some examples, the non-RT RIC 168 or the near-RT RIC 164 may be configured to tune RAN behavior or performance. For example, the non-RT RIC 168 may monitor long-term trends and patterns of performance and employ AI / ML models to perform corrective actions through the SMO framework 166 (such as via reconfiguration of O1) or via creation of RAN management policies (such as A1 policies).

[0070] Figure 2 is a component block diagram illustrating an example computing and wireless modem system 200 suitable for implementing any of the various embodiments. The various embodiments may be implemented on a number of single-processor and multi-processor computer systems, including system-on-chip (SoC) or system-in-package.

[0071] refer to Figure 1A to Figure 2 , the illustrated example wireless device 200 (which may be a system-in-package in some embodiments) includes two SoCs 202, 204 coupled to a clock 206, a voltage regulator 208, at least one subscriber identity module (SIM) 268 and / or a SIM interface and a wireless transceiver 266, which is configured to send and receive wireless communications to / from a network wireless device such as a base station 110a via an antenna (not shown). In some embodiments, the first SoC 202 operates as a central processing unit (CPU) of the wireless device, which executes the instructions of the software application by performing arithmetic, logic, control, and input / output (I / O) operations specified by the instructions of the software application. In some embodiments, the second SoC 204 may operate as a dedicated processing unit. For example, the second SoC 204 may operate as a dedicated 5G processing unit responsible for managing high-capacity, high-speed (e.g., 5 Gbps, etc.) and / or very high frequency short wavelength (e.g., 28 GHz millimeter wave spectrum, etc.) communications.

[0072] The first SoC 202 may include a digital signal processor (DSP) 210, a modem processor 212, a graphics processor 214, an application processor (AP) 216, one or more coprocessors 218 (e.g., vector coprocessors) connected to one or more of these processors, memory 220, custom circuits 222, system components and resources 224, interconnects / bus modules 226, one or more temperature sensors 230, a thermal management unit 232, and a thermal power envelope (TPE) component 234. The second SoC 204 may include a 5G modem processor 252, a power management unit 254, an interconnect / bus module 264, multiple mmWave transceivers 256, memory 258, and various additional processors 260 (such as an application processor, a packet processor, etc.).

[0073] Each processor 210, 212, 214, 216, 218, 252, 260 may include one or more cores, and each processor / core may perform operations independently of other processors / cores. For example, the first SoC 202 may include a processor that executes a first type of operating system (e.g., FreeBSD, LINUX, OS X, etc.) and a processor that executes a second type of operating system (e.g., MICROSOFT WINDOWS10). In addition, any or all of the processors 210, 212, 214, 216, 218, 252, 260 may be included as part of a processor cluster architecture (e.g., a synchronous processor cluster architecture, an asynchronous or heterogeneous processor cluster architecture, etc.).

[0074] The first SoC 202 and the second SoC 204 may include various system components, resources, and custom circuits for managing sensor data, analog-to-digital conversion, wireless data transmission, and for performing other specialized operations, such as decoding data packets and processing encoded audio and video signals for rendering in a web browser. For example, the system components and resources 224 of the first SoC 202 may include power amplifiers, voltage regulators, oscillators, phase-locked loops, peripheral bridges, data controllers, memory controllers, system controllers, access ports, timers, and other similar components used to support processors and software clients running on the wireless device. The system components and resources 224 and / or custom circuits 222 may also include circuits for interfacing with peripheral devices (such as cameras, electronic displays, wireless communication devices, external memory chips, etc.).

[0075] The first SoC 202 and the second SoC 204 may communicate via an interconnect / bus module 250. The various processors 210, 212, 214, 216, 218 may be interconnected to one or more memory elements 220, system components and resources 224, and custom circuits 222, and a thermal management unit 232 via an interconnect / bus module 226. Similarly, the processor 252 may be interconnected to a power management unit 254, a millimeter wave transceiver 256, a memory 258, and various additional processors 260 via an interconnect / bus module 264. The interconnect / bus modules 226, 250, 264 may include an array of reconfigurable logic gates and / or implement a bus architecture (e.g., CoreConnect, AMBA, etc.). Communications may be provided by an advanced interconnect, such as a high-performance network on chip (NoC).

[0076] The first SoC 202 and / or the second SoC 204 may also include an input / output module (not illustrated) for communicating with resources external to the SoC, such as a clock 206, a voltage regulator 208, one or more wireless transceivers 266, and at least one SIM 268 and / or a SIM interface (i.e., an interface for receiving one or more SIM cards). Resources external to the SoC (e.g., clock 206, voltage regulator 208) may be shared by two or more of the internal SoC processors / cores. At least one SIM 268 (or one or more SIM cards coupled to one or more SIM interfaces) may store information supporting multiple subscriptions (including a first 5G NR subscription and a second 5G NR subscription, etc.).

[0077] In addition to the example system-in-package 200 discussed above, various embodiments may be implemented in a wide variety of computing systems that may include a single processor, multiple processors, multi-core processors, or any combination thereof.

[0078] Figure 3 1 to 3 are component block diagrams illustrating a software architecture 300 including a radio protocol stack for a user plane and a control plane in wireless communications suitable for implementing any of the various embodiments. Figure 3, the wireless device 320 may implement the software architecture 300 to facilitate communication between the wireless device 320 (e.g., wireless devices 120a-120e, 200) and a base station 350 (e.g., base stations 110a-110d) of a communication system (e.g., 100). In some embodiments, the layers in the software architecture 300 may form a logical connection with the corresponding layers in the software of the base station 350. The software architecture 300 may be distributed between one or more processors (e.g., processors 212, 214, 216, 218, 252, 260). Although illustrated with respect to one radio protocol stack, in a multi-SIM (subscriber identity module) wireless device, the software architecture 300 may include multiple protocol stacks, each of which may be associated with a different SIM (e.g., in a dual SIM wireless communication device, two protocol stacks are associated with two SIMs, respectively). Although described below with reference to LTE communication layers, the software architecture 300 may support any of a variety of standards and protocols for wireless communication, and / or may include additional protocol stacks that support any of a variety of standards and protocols for wireless communication.

[0079] The software architecture 300 may include a non-access stratum (NAS) 302 and an access stratum (AS) 304. The NAS 302 may include functions and protocols that support packet filtering, security management, mobility control, session management, and services and signaling between a SIM of a wireless device and its core network 140. The AS 304 may include functions and protocols that support communication between a SIM and an entity (e.g., a base station) of a supported access network. Specifically, the AS 304 may include at least three layers (layer 1, layer 2, and layer 3), each of which may include various sublayers.

[0080] In the user plane and control plane, layer 1 (L1) of AS 304 may be a physical layer (PHY) 306, which may oversee functions that implement transmission and / or reception on the air interface. Examples of such physical layer 306 functions may include cyclic redundancy check (CRC) attachment, decoding blocks, scrambling and descrambling, modulation and demodulation, signal measurement, MIMO, etc. The PHY layer 306 may include various logical channels, including a physical downlink control channel (PDCCH) and a physical downlink shared channel (PDSCH). As an example, the PHY layer 306 may support CSI measurement and reporting (e.g., CQI measurement and reporting).

[0081] In the user plane and the control plane, layer 2 (L2) of AS 304 may be responsible for the link between the wireless device 320 and the base station 350 on the physical layer 306. In various embodiments, layer 2 may include a medium access control (MAC) sublayer 308, a radio link control (RLC) sublayer 310, and a packet data convergence protocol (PDCP) 312 sublayer, each of which forms a logical connection that terminates at the base station 350.

[0082] In the control plane, layer 3 (L3) of AS 304 may include a radio resource control (RRC) sublayer 3. Although not shown, software architecture 300 may include additional layer 3 sublayers, as well as various upper layers above layer 3. In various embodiments, RRC sublayer 313 may provide functionality including broadcasting system information, paging, and establishing and releasing RRC signaling connections between wireless devices 320 and base stations 350.

[0083] In various embodiments, the PDCP sublayer 312 may provide uplink functions including multiplexing between different radio bearers and logical channels, sequence number addition, handover data handling, integrity protection, encryption, and header compression. In the downlink, the PDCP sublayer 312 may provide functions including in-sequence delivery of data packets, duplicate data packet detection, integrity verification, decryption, and header decompression.

[0084] In the uplink, the RLC sublayer 310 may provide segmentation and concatenation of upper layer data packets, retransmission of lost data packets, and automatic repeat request (ARQ). In the downlink, the RLC sublayer 310 functions may include reordering of data packets to compensate for out-of-order reception, reassembly of upper layer data packets, and ARQ.

[0085] In the uplink, the MAC sublayer 308 may provide functions including multiplexing between logical and transport channels, random access procedures, logical channel priorities, and hybrid ARQ (HARQ) operations. In the downlink, MAC layer functions may include channel mapping within a cell, demultiplexing, discontinuous reception (DRX), and HARQ operations.

[0086] While the software architecture 300 may provide functionality for sending data over a physical medium, the software architecture 300 may further include at least one host layer 314 to provide data transfer services to various applications in the wireless device 320. In some embodiments, the application-specific functionality provided by the at least one host layer 314 may provide an interface between the software architecture and a general-purpose processor.

[0087] In other embodiments, the software architecture 300 may include one or more higher logical layers (e.g., transport, session, presentation, application, etc.) that provide host layer functionality. In some embodiments, the software architecture 300 may include an application layer, where a logical connection terminates at another device (e.g., an end-user device, a server, etc.). In some embodiments, the software architecture 300 may also include a hardware interface 316 between the physical layer 306 and communication hardware (e.g., one or more radio frequency (RF) transceivers) in the AS 304.

[0088] Figure 41 to 2 is a component block diagram illustrating an example system 400 configured to establish secure device-to-device communications between two wireless devices according to some embodiments. Figure 4 , the system 400 may include one or more wireless devices 402 (e.g., wireless devices 102, 104, 202, 320) and external resources 418 (e.g., AR glasses, AR server), which may communicate via a wireless communication network 424. The external resources 418 may include information sources external to the system 400, external entities participating with the system 400, or other resources. In some implementations, some or all of the functionality attributed herein to the external resources 418 may be provided by resources included in the system 400. The system 400 may include multiple hardware, software, and / or firmware components that operate together to provide the functionality attributed herein to the processor 422.

[0089] The wireless device 402 may include an electronic storage device 420, which may be configured to store information related to the functions implemented by the session initiation protocol (SIP) setup module 430, the send-receive module 432, the interface display module 434, the AR interface module 436, the synthesizer module 438, the encoding / decoding module 440, the RTP timestamp module 442, the coordinate converter module 444, the image capture module 446 and any other instruction modules.

[0090] Electronic storage 420 may include non-transitory storage media that electronically stores information. Electronic storage 420 may include one or both of a system storage device that is provided integrally with system 200 (i.e., substantially non-removable) and / or a removable storage device that is removably connected to system 200 via, for example, a port (e.g., a universal serial bus (USB) port, a firewire port, etc.) or a drive (e.g., a disk drive, etc.). In various embodiments, electronic storage 420 may include one or more of charge-based storage media (e.g., EEPROM, RAM, etc.), solid-state storage media (e.g., flash drives, etc.), optically readable storage media (e.g., optical disks, etc.), magnetically readable storage media (e.g., magnetic tape, magnetic hard drives, floppy disk drives, etc.), and / or other electronically readable storage media. Electronic storage 420 may include one or more virtual storage resources (e.g., cloud storage, virtual private networks, and / or other virtual storage resources). Electronic storage 420 may store software algorithms, information determined by processor 422, and / or other information that enables system 200 to operate as described herein.

[0091] The wireless device 402 may be configured by machine-readable instructions 406. The machine-readable instructions 406 may include one or more instruction modules. The instruction modules may include computer program modules. The instruction modules may include one or more of a SIP setup module 430, a send-receive module 432, an interface display module 434, an AR interface module 436, a synthesizer module 438, an encoding / decoding module 440, an RTP timestamp module 442, a coordinate converter module 444, an image capture module 446, and other instruction modules (not illustrated). The wireless device 402 may include a processor 422 configured to implement the machine-readable instructions 406 and corresponding modules.

[0092] Processor 422 may include one of a plurality of local processors that may be configured to provide information processing capabilities in system 200. Thus, processor 422 may include one or more of the following: a digital processor, an analog processor, a digital circuit designed to process information, an analog circuit designed to process information, a state machine, and / or other mechanisms for electronically processing information. Although processor 422 may be configured to provide information processing capabilities in system 200, processor 422 may include one or more of the following: a digital processor, an analog processor, a digital circuit designed to process information, an analog circuit designed to process information, a state machine, and / or other mechanisms for electronically processing information. Figure 4 4. In some embodiments, processor 422 may include multiple processing units. These processing units may be physically located within the same device, or processor 422 may represent processing functionality of multiple devices distributed in system 200.

[0093] In some embodiments, the processor 422 executing the SIP setup module 430 may be configured to establish a dual video call including a first data stream and a second data stream with the sending wireless device in accordance with RFC 4796. In some embodiments, the processor 422 executing the SIP setup module 430 may be configured to configure a modem of the receiving wireless device to receive data via the first data stream and the second data stream based on the call setup request. In some embodiments, the processor 422 executing the SIP setup module 430 may be configured to configure a modem of the sending wireless device to send data via the first data stream and the second data stream based on the call setup request.

[0094] In some embodiments, the processor 422 executing the send-receive module 432 may be configured to receive real image data from the sending wireless device via a first data stream. In some embodiments, the processor 422 executing the send-receive module 432 may be configured to receive AR content from the sending wireless device via a second data stream as an ALT video stream, wherein the AR content is used to be displayed together with the real image data. In some embodiments, the processor 422 executing the send-receive module 432 may be configured to send real image data to the receiving wireless device via a first data stream. In some embodiments, the processor 422 executing the send-receive module 432 may be configured to send AR content to the receiving device via a second data stream as an ALT video stream, wherein the AR content is used to be displayed together with the real image data. In some embodiments, the processor 422 executing the send-receive module 432 may be configured to receive a call establishment request from the sending wireless device. In some embodiments, the processor 422 executing the send-receive module 432 may be configured to receive encoded real image data from the sending wireless device via a first data stream. In some embodiments, the processor 422 executing the send-receive module 432 may be configured to receive encoded AR content from the sending wireless device via a second data stream. In some embodiments, the processor 422 executing the send-receive module 432 may be configured to send real image data and coordinate transformations to a monitor communicatively coupled to the receiving wireless device so that the monitor displays the real image data and coordinate transformations. In some embodiments, the processor 422 executing the send-receive module 432 may be configured to send AR content and coordinate transformations to an AR user equipment communicatively coupled to the receiving wireless device so that the AR user equipment displays the AR content and coordinate transformations. In some embodiments, the processor 422 executing the send-receive module 432 may be configured to send a synthetic 2D image to an AR user equipment communicatively coupled to the receiving wireless device so that the AR user equipment displays the synthetic 2D image. In some embodiments, the processor 422 executing the send-receive module 432 may be configured to receive a synthetic 2D image synthesized by AR content and real image data from the sending wireless device via a second data stream.

[0095] In some embodiments, the processor 422 of the execution interface display module 434 may be configured to display the AR content in combination with the real image data. In some embodiments, the processor 422 of the execution interface display module 434 may be configured to display the decoded AR content in combination with the decoded real image data. In some embodiments, the processor 422 of the execution interface display module 434 may be configured to display the synthesized 2D image on the display terminal of the receiving wireless device. In some embodiments, the processor 422 of the execution interface display module 434 may be configured to display the synthesized 2D image

[0096] In some embodiments, the processor 422 executing the AR interface module 436 may be configured to communicate with the transmit-receive module so that the AR interface module 436 performs display processing in a manner similar to the interface display module 434 .

[0097] In some embodiments, the processor 422 executing the synthesizer module 438 may be configured to synthesize the AR content with the real image data to form a synthesized 2D image.

[0098] In some embodiments, the processor 422 executing the encoding / decoding module 440 may be configured to decode the encoded real image data to obtain decoded real image data. In some embodiments, the processor 422 executing the encoding / decoding module 440 may be configured to decode the encoded AR content to obtain decoded AR content. In some embodiments, the processor 422 executing the encoding / decoding module 440 may be configured to encode the real image data to obtain the encoded real image data. In some embodiments, the processor 422 executing the encoding / decoding module 440 may be configured to encode the AR content to obtain the encoded AR content.

[0099] In some embodiments, the processor 422 executing the RTP timestamp module 442 may be configured to correlate the first data stream and the second data stream with a real-time transport protocol (RTP) timestamp according to RFC3550.

[0100] In some embodiments, the processor 422 executing the coordinate converter module 444 may be configured to generate a coordinate transformation for the axial and / or spatial orientation of the display device relative to the perspective of the real-world image.

[0101] In some embodiments, processor 422 executing image capture module 446 may be configured to capture real-world images from the user's or camera's perspective.

[0102] The description of the functionality provided by different modules 430-446 is for illustrative purposes and is not intended to be limiting, as any of the modules 430-446 may provide more or less functionality than described. For example, one or more of the modules 430-446 may be eliminated, and some or all of their functionality may be provided by other modules in the modules 430-446. As another example, the processor 422 may execute one or more additional modules that may perform some or all of the functionality attributed below to one of the modules 430-446.

[0103] Processor 422 may execute modules 430 - 446 and / or other modules via software, hardware, firmware, or some combination of software, hardware, and / or firmware and / or other mechanisms for configuring processing capabilities on processor 422 .

[0104] Figure 5 is a message flow diagram 500 illustrating the exchange of messages between a sending wireless device 502, a network 140 (eg, a SIP proxy), and a receiving wireless device 504 during call setup and an active call according to various embodiments. Figures 1A to 5 , the sending wireless device 502 can establish an RTP / RTCP-based dual video call including the MAIN data stream and the ALT data stream with the receiving wireless device 504.

[0105] In some embodiments, the telecommunication network protocol (e.g., SIP) may be modified to include additional protocol messages to determine the capabilities of the receiving wireless device 504 and to establish a call between the sending wireless device 502 and the receiving wireless device 504 based on the determined capabilities. In some embodiments, standardized messages (e.g., INVITE, ACK, etc.) of the telecommunication network protocol (e.g., SIP) may be modified to include additional features for determining the capabilities of the receiving wireless device 504 and to establish a call between the sending wireless device 502 and the receiving wireless device 504 based on the determined capabilities.

[0106] In the illustrated example, the telecommunication network implements SIP. However, additional protocol messages may be implemented in a similar manner in various other telecommunication network protocols before a connection is established between the calling wireless device and the called wireless device. In a telecommunication network implementing SIP, additional protocol messages may be introduced into sequence S501 before the sending wireless device 502 and the receiving wireless device 504 enter a data exchange phase (e.g., a dual video IP Multimedia Subsystem (IMS) call; RTP / RTCP stream). Figure 5 As illustrated, various wireless devices may communicate with each other via telecommunications network 140 .

[0107] The transmitting wireless device 502 may determine whether the receiving wireless device 504 is capable of receiving the AR content through the auxiliary stream in the RTP / RTCP configuration. If the transmitting wireless device 502 determines (e.g., based on a response (e.g., 200OK) from the receiving wireless device 504) that the receiving wireless device 504 is capable of receiving the AR content through the auxiliary stream in the RTP / RTCP configuration, the transmitting wireless device 502 may configure the modem of the transmitting wireless device 502 to prepare for sending the AR content through the auxiliary / ALT stream in addition to causing the receiving wireless device 504 to configure the modem of the receiving wireless device 504 to prepare for receiving the AR content through the auxiliary / ALT stream.

[0108] The sending wireless device 502 may determine whether the receiving wireless device 504 is capable of receiving AR content via an auxiliary stream in an RTP / RTCP configuration, and may indicate to the receiving wireless device 504 a request to initialize an auxiliary stream for delivering AR information by including additional information within a SIP message (such as in a message header and / or a SIP OPTIONS message).

[0109] In some embodiments, a user agent client (UAC) (e.g., a sending wireless device 502) may implement a SIP User-Agent header field, and a user agent server (UAS) (e.g., a receiving wireless device 504) may implement a SIP Server header field. The SIP User-Agent header field may include information about the sending wireless device 502 that initiated the request. The SIP User-Agent header field may be included as part of an INVITE, PRACK, UPDATE, and / or ACK message, and may include information indicating that the sending wireless device 502 is capable of and / or is requesting that AR content be communicated over an auxiliary stream. The SIP Server header field may include information about software implemented by the receiving wireless device 504 for processing requests from the sending wireless device. The SIP Server header field may be included as part of a SESSION PROGRESS, PRACK OK, UPDATE OK, RING, and / or OKINVITE message, and may include information indicating that the receiving wireless device 504 is capable of and / or has been configured to communicate AR content over an auxiliary stream.

[0110] In some embodiments, a Contact header field (e.g., RFC 3840) may be used to indicate the characteristics and device capabilities of the sending wireless device 502 and the receiving wireless device 504. The sending wireless device 502 may implement the Contact header field within the INVITE, UPDATE, and / or INVITE OK messages to indicate that the sending wireless device 502 is capable of and / or is requesting to communicate AR content via the auxiliary stream. The receiving wireless device 504 may implement the Contact header field within the SESSION PROGRESS, UPDATE OK, and / or RINGING messages to indicate that the receiving wireless device 504 is capable of and / or has been configured to communicate AR content via the auxiliary stream.

[0111] In some embodiments, a SIP OPTIONS message and a corresponding 200 OK (OPTIONS) message may be generated and / or configured to allow the sending wireless device 502 and the receiving wireless device 504 to query each other's device capabilities (e.g., the ability to stream AR content via an auxiliary stream).

[0112] The receiving wireless device 504 may receive a call setup request from the sending wireless device 502. The request may be sent from the sending wireless device 502 to the receiving wireless device 504 according to any known standard, such as the Session Initiation Protocol (“SIP”). Based on the call setup request, the modem of the sending wireless device 502 may be configured to send data through a first data stream (i.e., a main data stream, MAIN data stream) and a second data stream (i.e., an auxiliary data stream, ALT data stream), and the modem of the receiving wireless device 504 may be configured to receive data through the first data stream and the second data stream. After the modems of the sending wireless device 502 and the receiving wireless device 504 are configured, a dual video call including the MAIN data stream and the ALT data stream may be established between the sending wireless device 502 and the receiving wireless device 504. The dual video call may be established and maintained according to a standard that implements multiple data streams within a single session, such as RFC 4796. After a dual video call is established between the sending wireless device 502 and the receiving wireless device 504 , various types of data may be communicated between the sending wireless device 502 and the receiving wireless device 504 in a communication S501 (eg, a session).

[0113] In some embodiments where the first data stream and the second data stream are RTP streams (e.g., MAIN data stream and ALT data stream, respectively), the AR content may be communicated via the second data stream as part of an RTP extension header or otherwise embedded within an RTP extension header. In some embodiments where the first data stream and the second data stream are RTCP streams, the AR content may be communicated via the second data stream as part of an RTCP custom payload (e.g., an RTCP APP packet).

[0114] Figure 6 is a system block diagram illustrating an example AR communication system 600 that implements an ALT stream as an AR stream according to some embodiments. Figures 1A to 6 , the AR communication system 600 may include a transmitting wireless device 502, a receiving wireless device 504, an AR server 603, an AR UE 601, an AR UE 605, and a display terminal 607. The transmitting wireless device 502 may be communicatively coupled to the AR UE 601. The AR UE 601 and the AR UE 605 may be AR devices for rendering, displaying, or otherwise presenting AR content 611 to a user. For example, the AR UE 601 and the AR UE 605 may be AR glasses or AR headsets for rendering and displaying the AR content 611 to the user.

[0115] The transmitting wireless device 502 can perform direct device-to-device communication with the receiving wireless device 504 via wireless connection 624, perform direct device-to-device communication with the AR UE 601 via wireless connection 620, and perform direct device-to-device communication with the AR server 603 via wireless connection 622. The wireless connections 620, 622, and 624 can be any form of short-range wireless communication protocol, such as LTE-D, LTE side link (e.g., side link channel 124), WiFi, BT, BLE, or NFC, or can be any form of long-range communication (e.g., wireless communication link 122). In some embodiments, the transmitting wireless device 502 can communicate with the AR server 603 through a telecommunications network (e.g., network 140). The receiving wireless device 504 can perform direct device-to-device communication with the AR UE 605 via wireless connection 626. Wireless connection 626 may be any form of short range wireless communication protocol, such as LTE-D, LTE sidelink (e.g., sidelink channel 124), WiFi, BT, BLE, or NFC, or may be any form of long range communication (e.g., wireless communication link 122).

[0116] As illustrated, the sending wireless device 502 (e.g., a mobile phone) can establish a dual video IP Multimedia Subsystem (IMS) call with the receiving wireless device 504 (e.g., a mobile phone) according to transactions T1 and T2, and establish a session according to SIP procedures (e.g., RFC 4796), as described in reference Figure 5 The session between the transmitting wireless device 502 and the receiving wireless device 504 may be established after determining that the receiving wireless device 504 is capable of receiving data through multiple data streams (eg, MAIN, ALT streams) in a dual video RFC4796 configuration.

[0117] The AR UE 601 may be configured to capture real-world data, such as snapshots or real-world images 612, based on the view of the real-world perspective 610. A user operating the transmitting wireless device 502 and the AR UE 601 tethered or otherwise connected to the transmitting wireless device 502 may perceive a view of an object, creature, landscape, or any other type through the AR UE 601. The AR UE 601 may capture an image, video, or any sequence of images of the view perceived by the user operating the AR UE 601. In communication S602, the AR UE 601 may transmit one or more captured real-world images 612 to the transmitting wireless device 502.

[0118] In communication S603, the sending UE may send the real-world image 612 and / or metadata derived from the real-world image 612 to the AR server 603. In some implementations, the sending wireless device 502 may analyze the real-world image 612 received from the AR UE 601 to obtain metadata describing the real-world image 612. In such implementations, the sending wireless device 502 may send the obtained metadata to the AR server 603. In some implementations, the sending wireless device 502 may send the real-world image 612 received from the AR UE 601 directly to the AR server 603, and the AR server 603 may analyze the real-world image 612 to obtain metadata corresponding to the real-world image 612.

[0119] In communication S604, the AR server 603 may determine or otherwise generate AR content 611 based on metadata derived from the real-world image 612, and may send the AR content 611 data to the transmitting wireless device 502. The AR content 611 may include graphic content to be applied or otherwise enhanced on the real-world image 612. In communication S605, the transmitting wireless device 502 may send the AR content 611 data to the AR UE 601, and the AR UE 601 may display the AR content 611 to the user (e.g., on eyeglasses, on a headset screen).

[0120] In communication S501a, the sending wireless device 502 may send the real-world image 612 and / or any real-world image data associated with the real-world image 612 to the receiving wireless device 504 via the main stream. The sending wireless device 502 may encode the real-world image 612 data and may send the encoded real-world image 612 data to the receiving wireless device 504. In an embodiment implementing RTP, the real-world image 612 data may be sent as part of an RTP extension header. In an embodiment implementing RTCP, the real-world image 612 data may be sent as part of a custom payload (e.g., an RTCP APP packet).

[0121] In communication S501b, the sending wireless device 502 may send AR content 611 data to the receiving wireless device 504 via an auxiliary stream. The sending wireless device 502 may encode the AR content 611 and may send the encoded AR content 611 data to the receiving wireless device 504. In an implementation scheme that implements RTP, the AR content 611 data may be sent as part of an RTP extension header. In an implementation scheme that implements RTCP, the AR content 611 data may be sent as part of a custom payload (e.g., an RTCP APP packet). The auxiliary stream may be configured based on one or more query messages and responses implemented by the sending wireless device 502 and the receiving wireless device 504, such as a SIP OPTIONS message and / or other SIP messages including a SIP user agent header field, a SIP server header field, and / or a contact header field that define device capabilities. The auxiliary stream may be configured to include a payload type, port, codec, and other format parameters that allow the sending wireless device 502 and the receiving wireless device 504 to distinguish the auxiliary stream from the mainstream.

[0122] In some embodiments, the transmission of the real-world image 612 data via the main stream and the AR content 611 data via the auxiliary data stream from the transmitting wireless device 502 to the receiving wireless device 504 may occur before, after, or during the transmission of the AR content 611 data from the transmitting wireless device 502 to the AR UE 601. For example, the transmitting wireless device 502 may receive the AR content 611 data from the AR server 603, and may prioritize the transmission of the real-world image 612 data and the AR content 611 data to the receiving wireless device 504 to minimize the time between the generation of the AR content 611 data and the receipt of the AR content 611 data by the receiving wireless device.

[0123] In some embodiments, the transmission of AR content 611 data from the transmitting wireless device 502 to the receiving wireless device 504 via the auxiliary data stream may occur before, after, or during the transmission of the real-world image 612 data via the primary data stream. An RTP timestamp may be generated and applied to the real-world image 612 data and the AR content 611 data so that each set of real-world image 612 data and corresponding AR content 611 data may share the same timestamp or may have a timestamp indicating that a particular AR content 611 is associated with a particular real-world image 612. Thus, if any data via the primary and auxiliary streams is received out of order, or if AR content 611 data corresponding to a real-world image 612 is received by the receiving wireless device 504 before or after the corresponding real-world image 612 data is received, the receiving wireless device 504 may correctly match the received AR content 611 data to the corresponding received real-world image 612 data in order to render the AR content 611 in conjunction with the real-world image 612. The timestamps of the AR content 611 and the real-world image 612 may be assigned and associated based on RFC 3550.

[0124] In embodiments where the AR content 611 data and the real-world image 612 data have been encoded by the transmitting wireless device 502, the receiving wireless device 504 may decode the encoded AR content 611 data and the encoded real-world image 612 data to obtain decoded AR content 611 data and decoded real-world image 612 data that may be used to render an AR environment to a user of the receiving wireless device 504. The receiving wireless device 504 may decode the encoded AR content 611 data and the encoded real-world image 612, and may perform operations such as coordinate conversions to prepare for rendering the decoded AR content 611 data and the decoded real-world image 612 data on a display.

[0125] In communication S608, the receiving wireless device 504 may send the decoded real-world image 612 data (i.e., if previously encoded) to the display terminal 607. The display terminal 607 may be any terminal, interface, or screen capable of presenting a user with a view of the real-world image 612. The display terminal 607 may convert the real-world image 612 data to display the real-world image 612 to the user operating the receiving wireless device 504 and the AR UE 605.

[0126] In communication S609, the receiving wireless device 504 may transmit the decoded AR content 611 data (i.e., if previously encoded) to the AR UE 605. The AR UE 605 may convert the AR content 611 data to render the AR content 611 to the user, wherein the AR content 611 corresponds to the real-world image 612. Thus, the user operating the AR UE 605 may view the display terminal 607 through the AR UE 605, and the AR content 611 corresponding to the real-world image 612 may be displayed on the AR UE 605 such that the AR content 611 is overlaid on the real-world image 612 according to the viewing angle of the user.

[0127] In some embodiments, the receiving wireless device 504 may include a converter that can provide coordinate transformations to the display terminal 607 and the AR UE 605. The coordinate transformation can provide recommendations to the user to adjust the position and angle of the AR UE 605 in three-dimensional (3D) space so that the AR content 611 can be appropriately and accurately superimposed on the corresponding real-world image 612 displayed on the display terminal 607. In some embodiments, the real-world image 612 data and the AR content 611 received by the receiving wireless device 504 can include coordinate transformation information, and the AR UE 605 and the display terminal 607 can use the coordinate transformation information to generate a recommended axial and spatial orientation of the AR UE 605 relative to the display terminal 607 and display it to the user.

[0128] Figure 7 is a system block diagram illustrating an example communication system that implements an ALT stream as an AR stream according to some embodiments. Unless otherwise specified herein, Figure 7 The components illustrated in the reference Figure 6 The same components described above function in the same or similar manner. For example, the operations and processes performed by the transmitting wireless device 502, the receiving wireless device 504, the AR server 603, the AR UE 601, and the AR UE 605 may be described with reference to Figure 6 A similar approach as described is performed to deliver AR content 611 data via the ALT stream so as to display the AR content 611 in conjunction with the real-world image 612 .

[0129] In some embodiments, the receiving wireless device 504 may include a synthesizer 700, which may be a software function block, program, or mechanism implemented by a processor of the receiving wireless device 504. After receiving the AR content 611 data and the real-world image 612 data in communications S501a and S501b (and decoding the data if the data was previously encoded by the transmitting wireless device 502), the synthesizer 700 of the receiving wireless device 504 may correlate the AR content 611 data with the real-world image 612 data (e.g., via RTP timestamps). After the synthesizer 700 correlates the corresponding AR content 611 data with the real-world image 612 data, the synthesizer 700 may generate a synthesized two-dimensional (2D) image 702 in communication S701. The synthesized 2D image 702 may include the AR content 611 overlapped with the real-world image 612. In communication S702 , the receiving wireless device 504 may send the synthesized 2D image 702 to the AR UE 605 to enable the AR UE 605 to render and display the synthesized 2D image 702 to the user.

[0130] In some embodiments, the synthesizer 700 may be a component or functional block of a database or server that is communicatively coupled to the receiving wireless device 504. For example, the receiving wireless device 504 may decode the AR content 611 data and the real-world image 612 data and send it to an external server that implements the synthesizer 700, which may use the AR content 611 data and the real-world image 612 data to generate the synthesized 2D image 702, and the external server may send the synthesized 2D image 702 to the receiving wireless device 504 for display (i.e., on the receiving wireless device 504 and / or the AR UE 605).

[0131] Figure 8 is a system block diagram illustrating an example communication system that implements an ALT stream as an AR stream according to some embodiments. Figure 8 The components illustrated in the reference Figure 6 and Figure 7 The same components described above function in the same or similar manner. For example, the operations and processes performed by the transmitting wireless device 502, the receiving wireless device 504, and the AR server 603 may be as described with reference to Figure 6 and Figure 7 A similar approach as described is performed to deliver AR content 611 data via the ALT stream so as to display the AR content 611 in conjunction with the real-world image 612 .

[0132] In some embodiments, the transmitting wireless device 502 may include an image capture mechanism 502c (e.g., a mobile device camera) capable of generating a real-world image 612 based on the real-world perspective view 610. In communication S801, the image capture mechanism 502c may capture the real-world image 612. As previously described, the transmitting wireless device may send the real-world image 612 data to the AR server 603 in communication S603, and may receive the corresponding AR content 611 data in communication S604. Before, after, or during the processing of communications S501a and S501b, the transmitting wireless device 502 may optionally display the real-world image 612 in addition to the AR content 611 on the display 502d of the transmitting wireless device 502. After performing communications S501a and S501b to deliver the real-world image 612 data and the AR content 611 data to the receiving wireless device 504, the synthesizer 700 may correlate and combine the real-world image 612 and the AR content 611 in communication S802 to generate a synthesized 2D image 802. The receiving wireless device 504 may then display the synthesized image 802 on the display 504d of the receiving wireless device 504.

[0133] Fig. 9 is a system block diagram illustrating an example communication system that implements an ALT stream as an AR composite image stream according to some embodiments. Figure 8 The components illustrated in the reference Figures 6 to 8 The same components described above function in the same or similar manner. For example, the operations and processes performed by the transmitting wireless device 502, the receiving wireless device 504, and the AR server 603 may be as described with reference to Figures 6 to 8 A similar approach as described is performed to deliver AR synthetic image data via the ALT stream in order to display AR content 611 in conjunction with real-world image 612 .

[0134] In some embodiments, the transmitting wireless device 502 may include a synthesizer 900, which may be coupled to a reference Figure 7 and Figure 8 The synthesizer 700 described above performs operations in a similar manner. For example, in communication S902, the synthesizer 900 may receive (i.e., from the AR server 603) real-world image 612 data and AR content 611 data based on the real-world image 612 data, and may generate a synthesized 2D image 902 in which the AR content 611 is overlaid on the real-world image 612. In some embodiments, the transmitting wireless device 502 may display the synthesized 2D image 902 on the display 502d.

[0135] In communication S904, the transmitting wireless device 502 may transmit 2D synthetic image 902 data to the receiving wireless device 504 via the ALT stream or the auxiliary stream. The ALT stream may be preconfigured to transmit a synthetic 2D image including the synthetic 2D image 902. The receiving wireless device 504 may render the synthetic 2D image 902 received via the auxiliary stream by adjusting the rendering size of the synthetic 2D image 902 to fit the rendering screen of the display 504d (if any). The receiving wireless device 504 may then display the received synthetic 2D image 902 on the display 504d, and may also optionally display the real-world image 612. In some embodiments, the user of the receiving wireless device 504 may select which information (e.g., real-world image 612, synthetic 2D image 902) is displayed.

[0136] In view of the present disclosure, a person of ordinary skill in the art may envision additional implementations including any number of wireless devices (e.g., a transmitting wireless device 502, a receiving wireless device 504, an AR UE 601, an AR UE 605, an AR server 603, a display terminal 607, etc.) and combinations thereof. For example, a transmitting wireless device may include multiple modems, and the same AR content may be communicated to multiple receiving wireless devices via multiple ALT streams. For another example, any number of AR displays may be communicatively connected to a receiving wireless device so that multiple users may experience the same AR experience simultaneously. For another example, various implementations may be implemented with or without AR glasses, such that any transmitting wireless device or receiving wireless device involved in communicating AR content or synthesized 2D images via an ALT stream may additionally or alternatively display AR content and / or synthesized 2D images (i.e., instead of on AR glasses).

[0137] Fig. 10A is a process flow diagram of an example method 1000a that may be executed by a processor of a receiving wireless device for rendering AR content, according to various embodiments. FIG. 10B to FIG. 10E is a process flow diagram of example operations 1000b-1000e that may be performed as part of method 1000a for rendering augmented reality content, according to some embodiments. Figures 1A to 10E, the method 1000a and operations 1000b-1000e may be performed by a processor (e.g., 210, 212, 214, 216, 218, 252, 260, 422) of a wireless device (e.g., 120, 200, 320, 402, 504). In some embodiments, the processor (e.g., 210, 212, 214, 216, 218, 252, 260, 422) may be configured to perform operations via processor-executable instructions stored in a non-transitory processor-readable medium (e.g., 220, 258, 420). The means for performing the operations of the method 1000a and each of the operations 1000b-1000e may be a processor of the systems 100, 160, 200, 300, 400, 500, 600, 700, 800, and 900, such as reference 100, 160, 200, 300, 400, 500, 600, 700, 800, and 900. Figures 1A to 9 Processors 210, 212, 214, 216, 218, 252, 260, 422, etc. are described.

[0138] In block 1002, a processor of a receiving wireless device may perform operations including receiving real image data from a transmitting wireless device via a first data stream. The real image data or real-world image 612 data may be received by the receiving wireless device 504 from the transmitting wireless device 502 via a first data stream (e.g., MAIN data stream, main data stream). In some embodiments, receiving the real image data from the transmitting wireless device via the first data stream may include receiving encoded real image data from the transmitting wireless device via the first data stream. Means for performing the operations of block 1002 may include a processor (e.g., 210, 212, 214, 216, 218, 252, 260, 422) of a wireless device (e.g., 120, 200, 320, 402, 504) executing a transmit-receive module 432.

[0139] In block 1004, the processor of the receiving wireless device may perform operations including receiving AR content from the sending wireless device via a second data stream as an alternative video stream, wherein the AR content is used to be displayed together with the real image data. AR content data (e.g., AR content 611 data) may be received by the receiving wireless device (e.g., the receiving wireless device 504) from the sending wireless device (e.g., the sending wireless device 502) via a second data stream (e.g., ALT data stream, auxiliary data stream). In some embodiments, receiving AR content from the wireless device via the second data stream may include receiving encoded AR content from the sending wireless device via the second data stream. In some embodiments, receiving AR content from the sending wireless device via the second data stream may include receiving a synthetic 2D image synthesized by AR content and real image data from the sending wireless device via the second data stream. Means for performing the operations of block 1004 may include a processor (e.g., 210, 212, 214, 216, 218, 252, 260, 422) of a wireless device (e.g., 120, 200, 320, 402, 504) executing the transmit-receive module 432.

[0140] In some embodiments, the first data stream and the second data stream may be real-time transport protocol (RTP) streams, and the AR content may be received as part of an RTP extension header. In some embodiments, the first data stream and the second data stream may be real-time transport protocol (RTP) control protocol (RTCP) streams, and the AR content may be received as part of an RTCP custom payload (e.g., an RTCP APP packet).

[0141] In block 1006, the processor of the receiving wireless device may perform operations including displaying AR content (e.g., AR content 611) in conjunction with real image data (e.g., real-world image 612). After receiving the real image data and the AR content as described with reference to blocks 1002 and 1004, the processor may overlay the AR content on the real image data or may create a synthetic 2D image based on the AR content and the real image data to render an AR environment visible to a user of the receiving wireless device (e.g., receiving wireless device 504) on a display. In some embodiments, displaying the AR content in conjunction with the real image data may include displaying the decoded AR content in conjunction with the decoded real image data. In some embodiments, displaying the AR content in conjunction with the real image data may include: sending the real image data and the coordinate transformation to a monitor (e.g., display terminal 607) communicatively coupled to the receiving wireless device so that the monitor displays the real image data and the coordinate transformation, and sending the AR content and the coordinate transformation to an AR user equipment (e.g., AR UE 605) communicatively coupled to the receiving wireless device so that the AR user equipment displays the AR content and the coordinate transformation. In some embodiments, displaying the AR content in combination with the real image data may include displaying a synthetic 2D image on a display terminal of the receiving wireless device (e.g., 702, 802, 902). In some embodiments, displaying the AR content in combination with the real image data may include displaying the AR content in combination with the real image data based on an RTP timestamp. In some embodiments, displaying the AR content in combination with the real image data may include sending a synthetic 2D image to an AR user equipment communicatively coupled to the receiving wireless device so that the AR user equipment displays the synthetic 2D image. The components for performing the operations of box 1006 may include a processor (e.g., 210, 212, 214, 216, 218, 252, 260, 422) of a wireless device (e.g., 120, 200, 320, 402, 504) that executes the send-receive module 432, the interface display module 434, the AR interface module 436, and the coordinate converter module 444.

[0142] Fig. 10B Operations 1000b are illustrated that may be performed as part of method 1000a for rendering AR content according to some embodiments. Figures 1A to 10B , a processor of the receiving wireless device (e.g., the receiving wireless device 504) may perform operations including receiving a call setup request (e.g., a SIP request) from a sending wireless device (e.g., the sending wireless device 502) in block 1008. Means for performing the operations of block 1008 may include a processor (e.g., 210, 212, 214, 216, 218, 252, 260, 422) of a wireless device (e.g., 120, 200, 320, 402, 504) executing the send-receive module 432 and the SIP setup module 430.

[0143] In block 1010, a processor of the receiving wireless device may perform operations including configuring a modem (e.g., 212, 252) of the receiving wireless device to receive data via a first data stream and a second data stream based on the call setup request. Means for performing the operations of block 1010 may include a processor (e.g., 210, 212, 214, 216, 218, 252, 260, 422) of a wireless device (e.g., 120, 200, 320, 402, 504) executing the SIP setup module 430.

[0144] In block 1012, a processor of the receiving wireless device may perform operations including establishing a dual video call including a first data stream and a second data stream with the sending wireless device in accordance with RFC 4796. Means for performing the operations of block 1012 may include a processor (e.g., 210, 212, 214, 216, 218, 252, 260, 422) of a wireless device (e.g., 120, 200, 320, 402, 504) executing the SIP setup module 430.

[0145] After the operations in block 1012 , the processor may perform the operations in block 1002 as described.

[0146] Fig. 10C Operations 1000c are illustrated that may be performed as part of method 1000a for rendering AR content according to some embodiments. Figures 1A to 10C , after the operations in block 1004, in block 1014, a processor of the receiving wireless device (e.g., the receiving wireless device 504) may perform operations including decoding the encoded real image data (e.g., the real-world image 612 data) to obtain decoded real image data. Means for performing the operations of block 1014 may include a processor (e.g., 210, 212, 214, 216, 218, 252, 260, 422) of a wireless device (e.g., 120, 200, 320, 402, 504) executing the encoding / decoding module 440.

[0147] In block 1016, a processor of the receiving wireless device (e.g., the receiving wireless device 504) may perform operations including decoding the encoded AR content to obtain decoded AR content (e.g., AR content 611 data). Means for performing the operations of block 1016 may include a processor (e.g., 210, 212, 214, 216, 218, 252, 260, 422) of a wireless device (e.g., 120, 200, 320, 402, 504) executing the encoding / decoding module 440.

[0148] After the operations in block 1016 , the processor may perform the operations in block 1006 as described.

[0149] Fig. 10D Operations 1000d are illustrated that may be performed as part of method 1000a for rendering AR content according to some embodiments. FIG. 1A to FIG. 10D , after the operations in block 1004, in block 1018, a processor of the receiving wireless device (e.g., the receiving wireless device 504) may perform operations including associating the first data stream and the second data stream with a real-time transport protocol (RTP) timestamp in accordance with RFC 3550. Means for performing the operations of block 1018 may include a processor (e.g., 210, 212, 214, 216, 218, 252, 260, 422) of the wireless device (e.g., 120, 200, 320, 402, 504) executing the RTP timestamp module 442.

[0150] After the operations in block 1018, the processor may perform the operations in block 1006 as described.

[0151] Fig.10E Operations 1000e are illustrated that may be performed as part of method 1000a for rendering AR content according to some embodiments. Figures 1A to 10E , after the operations in block 1004, a processor of the receiving wireless device (e.g., the receiving wireless device 504) may perform operations including synthesizing the AR content with the real image data to form a synthesized 2D image in block 1020. Means for performing the operations of block 1020 may include a processor (e.g., 210, 212, 214, 216, 218, 252, 260, 422) of a wireless device (e.g., 120, 200, 320, 402, 504) executing the synthesizer module 438.

[0152] After the operations in block 1020 , the processor may perform the operations in block 1006 as described.

[0153] Fig.11 1 is a component block diagram of a network computing device 1100 such as a base station (e.g., base stations 110a-110d, 350) suitable for use with various embodiments. Such a network computing device (e.g., a base station such as a gNB, eNB, etc.) may include at least Fig.11 Referring to Figures 1 to Fig.11 , the network computing device 1100 may include a processor 1101 coupled to a volatile memory 1102 and a large-capacity non-volatile memory such as a disk drive 1103 .

[0154] The network computing device 1100 may also include a peripheral memory access device, such as a floppy disk drive, a compact disk (CD), or a digital video disk (DVD) drive 1106, coupled to the processor 1101. The network computing device 1100 may also include a network access port 1104 (or interface) coupled to the processor 1101 for establishing a data connection with a network, such as the Internet and / or a local area network coupled to other system computers and servers.

[0155] The network computing device 1100 may include one or more antennas 1107 for sending and receiving electromagnetic radiation, which may be connected to a wireless communication link. The network computing device 1100 may include additional access ports such as USB, FireWire, Thunderbolt, etc. for coupling to peripherals, external storage, or other devices.

[0156] Fig.12 is a component block diagram of a wireless device 1200 suitable for use with various embodiments. Fig.12 , various embodiments may be implemented on a variety of wireless devices 1200 (e.g., wireless devices 120, 200, 320, 402, 502, 504), examples of which are described in Fig.12 12 is illustrated in the form of a smart phone. The wireless device 1200 may include: a first SoC 202 (e.g., a SoC-CPU) coupled to a second SoC 204 (e.g., a SoC with 5G capabilities). The first SoC 202 and the second SoC 204 may be coupled to an internal memory 1216, a display 1212, and may be coupled to a speaker 1214. The first SoC 202 and the second SoC 204 may also be coupled to at least one SIM 268 and / or a SIM interface, which may store information supporting a first 5G NR subscription and a second 5G NR subscription, the first 5G NR subscription and the second 5G NR subscription supporting services on a 5G non-standalone (NSA) network.

[0157] Additionally, the wireless device 1200 may include an antenna 1204 for sending and receiving electromagnetic radiation, which may be connected to a wireless transceiver 266 coupled to one or more processors in the first SoC 202 and / or the second SoC 204. The wireless device 1200 may also include a menu selection button or rocker switch 1220 for receiving user input.

[0158] The wireless device 1200 also includes a sound coding / decoding (CODEC) circuit 1210 that digitizes the sound received from the microphone into data packets suitable for wireless transmission and decodes the received sound data packets to generate analog signals provided to the speaker to generate sound. In addition, one or more of the processors in the first SoC 202 and the second SoC 204, the wireless transceiver 266, and the CODEC 1210 may include a digital signal processor (DSP) circuit (not shown separately).

[0159] The processors of the wireless network computing device 1100 and the wireless device 1200 may be any programmable microprocessor, microcomputer, or one or more multi-processor chips that can be configured by software instructions (applications) to perform various functions, including the functions of the various embodiments described below. In some wireless devices, multiple processors may be provided, such as one processor dedicated to wireless communication functions within the SoC 204 and one processor dedicated to running other applications within the SoC 202. Software applications may be stored in the memory 220, 1216, which are then accessed and loaded into the processor. The processor may include internal memory sufficient to store application software instructions.

[0160] Specific implementation examples are described in the following paragraphs. Although some of the specific implementation examples in the following specific implementation examples are described in terms of example methods, other example implementations may include: example methods discussed in the following paragraphs implemented by a computing device, the computing device including a processor configured with processor-executable instructions to perform the operations of the methods of the following specific implementation examples; example methods discussed in the following paragraphs implemented by a computing device including components for performing the functions of the methods of the following specific implementation examples; and the example methods discussed in the following paragraphs may be implemented on a non-transitory processor-readable storage medium having processor-executable instructions stored thereon, the processor-executable instructions being configured to cause the processor of the computing device to perform the operations of the methods of the following specific implementation examples.

[0161] Embodiment 1. A method for rendering augmented reality content executed by a processor of a receiving wireless device, the method comprising: receiving real image data from a sending wireless device via a first data stream; receiving augmented reality (AR) content from the sending wireless device via a second data stream as an alternative video stream, wherein the AR content is used to be displayed together with the real image data, and displaying the AR content in combination with the real image data.

[0162] Embodiment 2. The method according to embodiment 1, wherein the first data stream and the second data stream are real-time transport protocol (RTP) streams, and wherein the AR content is received as part of an RTP extension header.

[0163] Embodiment 3. A method according to any one of Embodiments 1 or 2, wherein the first data stream and the second data stream are Real-time Transport Protocol (RTP) Control Protocol (RTCP) streams, and wherein the AR content is received as part of an RTCP application definition (APP) packet.

[0164] Embodiment 4. According to the method described in any one of Embodiments 1 to 3, the method further includes: receiving a call establishment request from the sending wireless device; configuring the modem of the receiving wireless device based on the call establishment request to receive data through the first data stream and the second data stream, and establishing a dual video call including the first data stream and the second data stream with the sending wireless device according to RFC 4796.

[0165] Embodiment 5. A method according to any one of Embodiments 1 to 4, wherein receiving real image data from the transmitting wireless device via the first data stream includes receiving encoded real image data from the transmitting wireless device via the first data stream, and wherein receiving AR content from the transmitting wireless device via the second data stream includes receiving encoded AR content from the transmitting wireless device via the second data stream, wherein the method further includes decoding the encoded real image data to obtain decoded real image data, and decoding the encoded AR content to obtain decoded AR content; and wherein displaying the AR content in combination with the real image data also includes displaying the decoded AR content in combination with the decoded real image data.

[0166] Embodiment 6. A method according to any one of embodiments 1 to 5, the method further comprising associating the first data stream and the second data stream with a real-time transport protocol (RTP) timestamp according to RFC 3550, wherein displaying the AR content in combination with the real image data comprises displaying the AR content in combination with the real image data based on the RTP timestamp.

[0167] Embodiment 7. A method according to any one of embodiments 1 to 6, wherein displaying the AR content in combination with the real image data includes: sending the real image data and the coordinate transformation to a monitor communicatively coupled to the receiving wireless device so that the monitor displays the real image data and the coordinate transformation, and sending the AR content and the coordinate transformation to an AR user equipment communicatively coupled to the receiving wireless device so that the AR user equipment displays the AR content and the coordinate transformation.

[0168] Embodiment 8. A method according to any one of embodiments 1 to 7, the method further comprising synthesizing the AR content with the real image data to form a synthetic two-dimensional (2D) image, wherein displaying the AR content in combination with the real image data comprises sending the synthetic 2D image to an AR user equipment communicatively coupled to the receiving wireless device so that the AR user equipment displays the synthetic 2D image.

[0169] Embodiment 9. According to the method described in any one of Embodiments 1 to 8, the method further includes synthesizing the AR content with the real image data to form a synthetic two-dimensional (2D) image, wherein displaying the AR content in combination with the real image data further includes displaying the synthetic 2D image on a display terminal of the receiving wireless device.

[0170] Embodiment 10. A method according to any one of embodiments 1 to 9, wherein receiving the augmented reality (AR) content from the transmitting wireless device via the second data stream includes receiving a synthetic two-dimensional (2D) image synthesized by the AR content and the real image data from the transmitting wireless device via the second data stream, and wherein displaying the AR content in combination with the real image data includes displaying the synthetic 2D image.

[0171] As used in this application, the terms "component", "module", "system", etc. are intended to include computer-related entities, such as but not limited to hardware, firmware, a combination of hardware and software, software, or software being executed, which are configured to perform specific operations or functions. For example, a component can be, but is not limited to, a process, a processor, an object, an executable, an execution thread, a program, and / or a computer running on a processor. By way of example, both an application running on a wireless device and a wireless device can be referred to as a component. One or more components may reside within a process and / or an execution thread, and a component may be located on a processor or core and / or distributed between two or more processors or cores. In addition, these components may be executed from various non-transient computer-readable media having various instructions and / or data structures stored thereon. Components can communicate via local and / or remote processes, function or procedure calls, electronic signals, data packets, memory reads / writes, and other known network, computer, processor, and / or process-related communication methods.

[0172] In the future, there will be a variety of different cellular and mobile communication services and standards available or under consideration, all of which can be implemented and benefit from various implementation schemes. Such services and standards include, for example, the Third Generation Partnership Project (3GPP), LTE systems, third generation wireless mobile communication technology (3G), fourth generation wireless mobile communication technology (4G), fifth generation wireless mobile communication technology (5G) and later generation 3GPP technologies, Global System for Mobile Communications (GSM), Universal Mobile Telecommunications System (UMTS), 3GSM, General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA) systems (e.g., cdmaOne, CDMA1020TM), Enhanced Data Rates for GSM Evolution (EDGE), Advanced Mobile Phone System (AMPS), Digital AMPS (IS-136 / TDMA), Evolution Data Optimized (EV-DO), Digital Enhanced Cordless Telecommunications (DECT), Worldwide Interoperability for Microwave Access (WiMAX), Wireless Local Area Networks (WLAN), Wi-Fi Protected Access I and II (WPA, WPA2), and Integrated Digital Enhanced Network (iDEN). Each of these technologies involves, for example, the transmission and reception of voice, data, signaling and / or content messages. It should be understood that any reference to terminology and / or technical details related to individual telecommunication standards or technologies is for illustrative purposes only and is not intended to limit the scope of the claims to a particular communication system or technology unless specifically recited in the claim language.

[0173] The various embodiments illustrated and described are provided merely as examples of various features of the claims. However, the features shown and described with respect to any given embodiment are not necessarily limited to the associated embodiment, and may be used or combined with other embodiments shown and described. Furthermore, the claims are not intended to be limited to any one example embodiment. For example, one or more operations of the method may replace or be combined with one or more operations of the method.

[0174] The foregoing method descriptions and process flow charts are provided only as illustrative examples, and are not intended to require or imply that the operations of the various embodiments must be performed in the order given. As will be appreciated by those skilled in the art, the order of operations in the foregoing embodiments may be performed in any order. Words such as "thereafter," "then," "next," etc. are not intended to limit the order of operations; these words are only used to guide the reader through the description of the method. In addition, any reference to a claim element in singular form (e.g., a reference using the article "one," "an," or "the") should not be interpreted as limiting the element to a singular.

[0175] The various exemplary logic blocks, modules, circuits, and algorithmic operations described in conjunction with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, various exemplary components, blocks, modules, circuits, and operations have been generally described above in terms of their functionality. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. Although the technical staff can implement the described functionality in different ways for each specific application, such specific implementation decisions should not be interpreted as departing from the scope of the claims.

[0176] The hardware for implementing the various exemplary logics, logic boxes, modules, and circuits described in conjunction with the embodiments disclosed herein can be implemented or executed with a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (TCUASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. Although a general-purpose processor can be a microprocessor, in an alternative, the processor can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Alternatively, some operations or methods can be performed by circuits specific to a given function.

[0177] In one or more embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, these functions may be stored as one or more instructions or codes on a non-transient computer-readable medium or a non-transient processor-readable medium. The operation of the method or algorithm disclosed herein may be embodied in a processor-executable software module, which may reside on a non-transient computer-readable or processor-readable storage medium. A non-transient computer-readable or processor-readable storage medium may be any storage medium accessible by a computer or processor. By way of example and not limitation, such non-transient computer-readable or processor-readable media may include RAM, ROM, EEPROM, flash memory, CD-ROM or other optical disk storage, disk storage or other magnetic storage device, or any other medium that can be used to store the desired program code in the form of an instruction or data structure and can be accessed by a computer. Disks and optical disks as used herein include compact discs (CDs), laser discs, optical disks, digital versatile discs (DVDs), floppy disks, and blue discs, wherein disks generally reproduce data magnetically, and optical disks reproduce data optically with lasers. The above combination is also included within the scope of non-transient computer-readable and processor-readable media. Additionally, the operations of a method or algorithm may reside as one or any combination or set of codes and / or instructions on a non-transitory processor-readable medium and / or computer-readable medium, which may be incorporated into a computer program product.

[0178] The above description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the claims. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the claims. Thus, the present disclosure is not intended to be limited to the embodiments shown herein, but should be accorded the broadest scope consistent with the following claims and the principles and novel features disclosed herein.

Claims

1. A method for rendering augmented reality content performed by a processor of a receiving wireless device, the method comprising: receiving real image data from a transmitting wireless device via a first data stream; receiving augmented reality (AR) content from the transmitting wireless device via a second data stream as an alternative video stream, wherein the AR content is for display with the real image data; as well as The AR content is displayed in combination with the real image data. 2 . The method of claim 1 , wherein the first data stream and the second data stream are Real-time Transport Protocol (RTP) streams, and wherein the AR content is received as part of an RTP extension header.

3. The method of claim 1, wherein the first data stream and the second data stream are Real-time Transport Protocol (RTP) Control Protocol (RTCP) streams, and wherein the AR content is received as part of an RTCP application definition (APP) packet.

4. The method according to claim 1, further comprising: receiving a call setup request from the sending wireless device; configuring a modem of the receiving wireless device to receive data via the first data stream and the second data stream based on the call setup request; as well as A dual video call including the first data stream and the second data stream is established with the sending wireless device according to RFC 4796.

5. The method according to claim 1, wherein: Receiving real image data from the transmitting wireless device via the first data stream further comprises receiving encoded real image data from the transmitting wireless device via the first data stream; Receiving AR content from the transmitting wireless device via the second data stream further comprises receiving encoded AR content from the transmitting wireless device via the second data stream; The method further comprises: decoding the encoded real image data to obtain decoded real image data; and decoding the encoded AR content to obtain decoded AR content; and Displaying the AR content in combination with the real image data further includes displaying the decoded AR content in combination with the decoded real image data.

6. The method according to claim 1, further comprising: correlating the first data stream and the second data stream with a Real-time Transport Protocol (RTP) timestamp according to RFC 3550, Wherein displaying the AR content in combination with the real image data further comprises displaying the AR content in combination with the real image data based on the RTP timestamp.

7. The method according to claim 1, wherein displaying the AR content in combination with the real image data further comprises: transmitting the real image data and the coordinate transformation to a monitor communicatively coupled to the receiving wireless device so that the monitor displays the real image data and the coordinate transformation; as well as The AR content and the coordinate transformation are sent to an AR user equipment communicatively coupled to the receiving wireless device, so that the AR user equipment displays the AR content and the coordinate transformation.

8. The method according to claim 1, further comprising: synthesizing the AR content with the real image data to form a synthetic two-dimensional (2D) image, Displaying the AR content in combination with the real image data further comprises sending the synthesized 2D image to an AR user equipment communicatively coupled to the receiving wireless device so that the AR user equipment displays the synthesized 2D image.

9. The method according to claim 1, further comprising: synthesizing the AR content with the real image data to form a synthetic two-dimensional (2D) image, Displaying the AR content in combination with the real image data further includes displaying the synthesized 2D image on a display terminal of the receiving wireless device.

10. The method of claim 1, wherein receiving the augmented reality (AR) content from the transmitting wireless device via the second data stream further comprises receiving a synthetic two-dimensional (2D) image synthesized by the AR content and the real image data from the transmitting wireless device via the second data stream, and Displaying the AR content in combination with the real image data further includes displaying the synthesized 2D image.

11. A wireless device, comprising: A processor configured with processor-executable instructions to: receiving real image data from a transmitting wireless device via a first data stream; receiving augmented reality (AR) content from the transmitting wireless device via a second data stream as an alternative video stream, wherein the AR content is for display with the real image data; as well as The AR content is displayed in combination with the real image data.

12. The wireless device of claim 11, wherein the first data stream and the second data stream are Real-time Transport Protocol (RTP) streams, and wherein the AR content is received as part of an RTP extension header.

13. The wireless device of claim 11, wherein the first data stream and the second data stream are Real-time Transport Protocol (RTP) Control Protocol (RTCP) streams, and wherein the AR content is received as part of an RTCP application definition (APP) packet.

14. The wireless device of claim 11, wherein the processor is configured with processor-executable instructions to perform operations further comprising: receiving a call setup request from the sending wireless device; configuring a modem of the receiving wireless device to receive data via the first data stream and the second data stream based on the call setup request; as well as A dual video call including the first data stream and the second data stream is established with the sending wireless device according to RFC 4796.

15. The wireless device of claim 11, wherein: The processor is configured with processor-executable instructions to perform operations causing: Receiving real image data from the transmitting wireless device via the first data stream further comprises receiving encoded real image data from the transmitting wireless device via the first data stream; Receiving AR content from the transmitting wireless device via the second data stream further comprises receiving encoded AR content from the transmitting wireless device via the second data stream; The processor is configured with processor-executable instructions to perform operations, the operations further comprising: decoding the encoded real image data to obtain decoded real image data; as well as decoding the encoded AR content to obtain decoded AR content; and The processor is configured with processor-executable instructions to perform operations such that displaying the AR content in conjunction with the real image data further comprises displaying the decoded AR content in conjunction with the decoded real image data.

16. The wireless device of claim 11, wherein the processor is configured with processor-executable instructions to perform operations further comprising correlating the first data stream and the second data stream with a Real-time Transport Protocol (RTP) timestamp in accordance with RFC 3550, The processor is configured with processor-executable instructions to perform operations such that displaying the AR content in combination with the real image data further comprises displaying the AR content in combination with the real image data based on the RTP timestamp.

17. The wireless device of claim 11, wherein the processor is configured with processor-executable instructions to perform operations such that displaying the AR content in conjunction with the real image data further comprises: transmitting the real image data and the coordinate transformation to a monitor communicatively coupled to the receiving wireless device so that the monitor displays the real image data and the coordinate transformation; as well as The AR content and the coordinate transformation are sent to an AR user equipment communicatively coupled to the receiving wireless device, so that the AR user equipment displays the AR content and the coordinate transformation.

18. The wireless device of claim 11, wherein the processor is configured with processor-executable instructions to perform operations further comprising synthesizing the AR content with the real image data to form a synthetic two-dimensional (2D) image, The processor is configured with processor-executable instructions to perform operations such that displaying the AR content in combination with the real image data further includes sending the synthesized 2D image to an AR user equipment communicatively coupled to the receiving wireless device so that the AR user equipment displays the synthesized 2D image.

19. The wireless device of claim 11, wherein the processor is configured with processor-executable instructions to perform operations further comprising synthesizing the AR content with the real image data to form a synthetic two-dimensional (2D) image, The processor is configured with processor-executable instructions to perform operations such that displaying the AR content in combination with the real image data further includes displaying the synthesized 2D image on a display terminal of the receiving wireless device.

20. The wireless device of claim 11, wherein the processor is configured with processor-executable instructions to perform operations that: Receiving the augmented reality (AR) content from the transmitting wireless device via the second data stream also includes receiving a synthetic two-dimensional (2D) image synthesized by the AR content and the real image data from the transmitting wireless device via the second data stream, and Displaying the AR content in combination with the real image data also includes displaying the synthesized 2D image.

21. A wireless device, comprising: means for receiving real image data from a transmitting wireless device via a first data stream; means for receiving augmented reality (AR) content from the transmitting wireless device via a second data stream that is an alternative video stream, wherein the AR content is for display with the real image data; and A component for displaying the AR content in combination with the real image data.

22. The wireless device according to claim 21, wherein any of the following conditions is met: The first data stream and the second data stream are Real-time Transport Protocol (RTP) streams, and wherein the AR content is received as part of an RTP extension header; or The first data stream and the second data stream are Real-time Transport Protocol (RTP) Control Protocol (RTCP) streams, and wherein the AR content is received as part of an RTCP Application Definition (APP) packet.

23. The wireless device of claim 21, further comprising: means for receiving a call setup request from said sending wireless device; means for configuring a modem of the receiving wireless device to receive data via the first data stream and the second data stream based on the call setup request; and Means for establishing a dual video call including the first data stream and the second data stream with the sending wireless device in accordance with RFC 4796.

24. The wireless device of claim 21, wherein: means for receiving real image data from the transmitting wireless device via the first data stream further comprising means for receiving encoded real image data from the transmitting wireless device via the first data stream; The means for receiving AR content from the transmitting wireless device via the second data stream also includes means for receiving encoded AR content from the transmitting wireless device via the second data stream; The wireless device further comprises: means for decoding the encoded real image data to obtain decoded real image data; and means for decoding the encoded AR content to obtain decoded AR content; and The means for displaying the AR content in combination with the real image data further includes means for displaying the decoded AR content in combination with the decoded real image data.

25. The wireless device of claim 21, further comprising: means for correlating said first data stream and said second data stream with a Real-time Transport Protocol (RTP) timestamp in accordance with RFC 3550, The component for displaying the AR content in combination with the real image data further includes a component for displaying the AR content in combination with the real image data based on the RTP timestamp.

26. The wireless device of claim 21, wherein the means for displaying the AR content in combination with the real image data further comprises: means for transmitting the real image data and the coordinate transformation to a monitor communicatively coupled to the receiving wireless device so that the monitor displays the real image data and the coordinate transformation; and Means for sending the AR content and the coordinate transformation to an AR user equipment communicatively coupled to the receiving wireless device so that the AR user equipment displays the AR content and the coordinate transformation.

27. The wireless device of claim 21, further comprising: means for synthesizing the AR content with the real image data to form a synthetic two-dimensional (2D) image, The means for displaying the AR content in combination with the real image data further comprises means for sending the synthesized 2D image to an AR user equipment communicatively coupled to the receiving wireless device so that the AR user equipment displays the synthesized 2D image.

28. The wireless device of claim 21, further comprising: means for synthesizing the AR content with the real image data to form a synthetic two-dimensional (2D) image, The component for displaying the AR content in combination with the real image data further includes a component for displaying the synthesized 2D image on a display terminal of the receiving wireless device.

29. The wireless device of claim 21, wherein the means for receiving the augmented reality (AR) content from the transmitting wireless device via the second data stream further comprises means for receiving a synthetic two-dimensional (2D) image synthesized by the AR content and the real image data from the transmitting wireless device via the second data stream, and The component for displaying the AR content in combination with the real image data also includes a component for displaying the synthetic 2D image.

30. A non-transitory processor-executable medium having stored thereon processor-executable instructions configured to cause a processor of a wireless device to perform operations comprising: receiving real image data from a transmitting wireless device via a first data stream; receiving augmented reality (AR) content from the transmitting wireless device via a second data stream as an alternative video stream, wherein the AR content is for display with the real image data; as well as The AR content is displayed in combination with the real image data.

Images