Data synchronization performed by user equipment related to connected device

Abstract

A method includes receiving, by a user equipment (UE), a first signal from a first device, the first signal including first data for presentation by a second device and second data for presentation by a third device, the first data and the second data being synchronized in time with each other. The UE extracts first data and second data from the first signal, and determines timings for transmitting a second signal including the first data to a second device and for transmitting a third signal including the second data to a third device, the timing is configured to cause presentation of the first data and the second data by the respective second and third devices to be approximately synchronized in time. The UE transmits a second signal to the second device based on the timing, and transmits a third signal to the third device based on the timing.

Description

Technical Field

[0001] Various example embodiments relate generally to wireless networks, and more particularly to synchronization of data performed by a user equipment (UE) related to devices connected to the UE. Background Art

[0002] Current technology allows user equipment (UE) to be connected to multiple peripheral devices. These devices can be paired or tethered to the UE via a wireless connection (such as the Institute of Electrical and Electronics Engineers (IEEE) 802.15 wireless standard connection). Because the processing (e.g., queuing, caching, etc.) delays and transmission delays in multiple peripheral devices may not be the same, data (e.g., media data, video stream data, and / or audio stream data) sent to one peripheral device connected to the UE may not be presented by the device in synchronization with data sent to another peripheral device connected to the UE. In addition, data received from individual devices to the UE may also be out of sync. Summary of the invention

[0003] In one aspect of the present disclosure, a method includes receiving, by a user equipment (UE), a first signal from a first device, wherein the first signal includes first data for presentation by a second device communicating with the UE and second data for presentation by a third device communicating with the UE, wherein the first data and the second data are synchronized with each other in time; extracting, by the UE, the first data from the first signal; extracting, by the UE, the second data from the first signal; determining a timing for sending a second signal including the first data to the second device and for sending a third signal including the second data to the third device, wherein the timing is configured to make presentation of the first data by the second device and presentation of the second data by the third device approximately synchronized in time; sending a second signal to the second device based on the timing; and sending a third signal to the third device based on the timing.

[0004] In one aspect of the method, the method may further include determining at least processing and transmission delays of the second device, wherein determining the timing may include determining the timing based on at least the processing and transmission delays of the second device.

[0005] In one aspect of the method, the method may further include determining processing and transmission delays of the third device, wherein determining the timing may include determining the timing based at least on the processing and transmission delays of the third device.

[0006] In one aspect of the method, sending the second signal to the second device based on timing may include sending the second signal to the second device at a first time, and sending the third signal to the third device based on timing may include sending the third signal to the third device at a second time different from the first time.

[0007] In one aspect of the method, the first time may precede the second time.

[0008] In one aspect of the method, the method may further include encoding the first presentation time offset in the second signal based on the timing.

[0009] In one aspect of the method, the first data may include a first data stream, and the second data may include a second data stream.

[0010] In one aspect of the method, the first data stream may be one of an audio data stream or a video data stream, and the second data is the other of the audio data stream and the video data stream.

[0011] In one aspect of the method, the method may further include decoding and processing the first data.

[0012] In one aspect of the method, the method may further include decoding and processing the second data.

[0013] In one aspect of the present disclosure, a method includes receiving, by a user equipment (UE), a first signal including first data from a first device communicating with the UE; receiving, by the UE, a second signal including second data from a second device communicating with the UE; determining, by the UE, a timing for synchronizing the first data and the second data in time; generating, by the UE, a third signal including the first data and the second data based on the timing, wherein the first data and the second data are synchronized in time in the third signal; and sending the third signal to a third device.

[0014] In one aspect of the method, determining the timing may include determining processing and transmission delays of the first device.

[0015] In one aspect of the method, determining the timing may include determining processing and transmission delays of the second device.

[0016] In one aspect of the method, generating the third signal may include mixing first data of the first signal with second data of the second signal into the third signal.

[0017] In one aspect of the method, generating the third signal may include synchronizing a presentation time of the first data with a presentation time of the second data in the third signal.

[0018] In one aspect of the method, synchronizing a presentation time of the first data with a presentation time of the second data in the third signal may include synchronizing data packets of the first data with data packets of the second data in time.

[0019] In one aspect of the method, the first data may include a first data stream, and the second data may include a second data stream.

[0020] In one aspect of the method, the first data stream may be an audio data stream and the second data may be a video data stream.

[0021] In one aspect of the method, the method may further include encoding and processing the first data.

[0022] In one aspect of the method, the method may further include encoding and processing the second data.

[0023] In one aspect of the present disclosure, a user equipment (UE) includes at least one processor and at least one memory storing instructions, which, when executed by the at least one processor, causes the user equipment to perform at least any one of the above methods.

[0024] In one aspect of the present disclosure, a processor-readable medium stores instructions that, when executed by at least one processor of a device, cause the device to perform at least any one of the above methods.

[0025] According to some aspects, the subject matter of the independent claims is provided. Some further aspects are defined in the dependent claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Some example embodiments will now be described with reference to the accompanying drawings.

[0027] Figure 1 is a diagram of an example embodiment of wireless networking between a network system and a user equipment (UE) including a plurality of devices connected to the UE according to one illustrative aspect of the present disclosure;

[0028] Figure 2 is a diagram of example components of a network system according to one illustrative aspect of the present disclosure;

[0029] Figure 3A is a diagram of an example embodiment of a UE, connected devices, and a network system showing example downlink communications according to one illustrative aspect of the present disclosure;

[0030] Figure 3B is a diagram of an example embodiment of a UE, connected devices, and a network system showing example uplink communications according to one illustrative aspect of the present disclosure;

[0031] Figure 4 is a diagram of an example embodiment of a UE, connected devices, and a network system showing combined downlink and uplink communications according to one illustrative aspect of the present disclosure;

[0032] Figure 5A and Figure 5B is a diagram of example signals and operations of a network system according to one illustrative aspect of the present disclosure;

[0033] Fig. 6A and Figure 6B is a diagram of example signals and operations of a network system according to another illustrative aspect of the present disclosure; and

[0034] Figure 7 is a diagram of an example embodiment of components of a UE, connected device, or network apparatus according to one illustrative aspect of the present disclosure. DETAILED DESCRIPTION

[0035] In the following description, certain specific details are set forth in order to provide a thorough understanding of the disclosed aspects. However, one skilled in the relevant art will recognize that the various aspects may be practiced without one or more of these specific details or using other methods, components, materials, etc. In other instances, well-known structures associated with transmitters, receivers, or transceivers are not shown or described in detail to avoid unnecessarily obscuring the description of the various aspects.

[0036] In this specification, reference to "one aspect" or "an aspect" indicates that a particular feature, structure, or characteristic described in connection with that aspect is included in at least one aspect. Thus, the appearance of the phrase "in one aspect" or "in an aspect" in various places in this specification does not necessarily refer to the same aspect. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more aspects.

[0037] The embodiments described in the present disclosure may be implemented in wireless network devices, such as, but not limited to, devices utilizing the Worldwide Interoperability for Microwave Access (WiMAX) TM ), Global System for Mobile Communications (GSM, 2G), GSM EDGE Radio Access Network (GERAN), General Packet Radio Service (GRPS), Universal Mobile Telecommunications System (UMTS, 3G) based on basic Wideband Code Division Multiple Access (W-CDMA), High Speed ​​Packet Access (HSPA), Long Term Evolution (LTE), Advanced LTE, Enhanced LTE (eLTE), 5G New Radio (5G NR), Advanced 5G, 6G (and beyond), and 802.11ax (Wi-Fi TM 6) and other wireless network systems. The term "eLTE" here means LTE evolution connected to a 5G core. LTE is also known as Evolved UMTS Terrestrial Radio Access (EUTRA) or Evolved UMTS Terrestrial Radio Access Network (EUTRAN).

[0038] The present disclosure may use the term "serving network device" to refer to a network node or network apparatus (or a portion thereof) that serves a UE. As used herein, the terms "send to," "receive from," and "cooperate with" (and variations thereof) include communications that may or may not involve communications through one or more intermediate devices or nodes. The term "acquire" (and variations thereof) includes first acquiring or reacquiring after a first acquisition. The term "connect" may refer to a physical connection or a logical connection.

[0039] The present disclosure uses 5G NR as an example of a wireless network, and may use a smartphone and / or an extended reality headset as an example of a UE. It should be understood that these examples are merely illustrative, and the present disclosure is applicable to other wireless networks and user equipment.

[0040] Figure 1 1 is a diagram depicting an example of wireless networking between a network system 100 and a user equipment (UE) 150, which includes multiple devices 160 (denoted as 160a and 160b) connected to the UE 150. The device 160 may be a peripheral device that is paired or tethered with the UE 150 via a wireless communication connection. For example, in various embodiments, the UE 150 and the device 160 may implement a radio technology (such as the Institute of Electrical and Electronics Engineers (IEEE) 802.15 radio technology) and be connected via the radio technology. However, those skilled in the art will appreciate that the device 160 and the UE 150 may be connected via other technologies.

[0041] Furthermore, the device 160 may include any type of device capable of being connected to the UE 150. For example, in various embodiments, the device 160a may be a headset device (which may also include a microphone) that is connected to the UE 150 to facilitate receiving and sending audio data signals from and to the UE 150. Furthermore, for example, according to an example embodiment, the device 160b may be a video display and / or video imaging device (e.g., augmented reality (AR), virtual reality (VR), or mixed reality (MR) glasses). Furthermore, for example, the device 160c may be a smart watch that may include video display and / or audio playback functionality, as well as video and / or video capture functionality.

[0042] The network system 100 may include one or more network nodes 120, one or more servers 110, and / or one or more network devices 130 (e.g., test equipment). The network node 120 will be described in more detail below. As used herein, the term "network device" may refer to any component of the network system 100, such as a server 110, a network node 120, a network device 130, any (multiple) components of the above, and / or any other (multiple) components of the network system 100. Examples of network devices include, but are not limited to, devices that implement various aspects of 5G NR, etc. The present disclosure describes embodiments related to 5G NR and embodiments involving various aspects defined by the Third Generation Partnership Project (3GPP). However, it is contemplated that embodiments related to other wireless network technologies are also included within the scope of the present disclosure.

[0043] The following description provides further details of an example of a network node. In a 5G NR network, a gNodeB (also referred to as a gNB) may include, for example, a node that provides NR user plane and control plane protocol terminations towards the UE and is connected to the 5G Core (5GC) via an NG interface, for example in accordance with 3GPP TS 38.300 V16.6.0 (2021-06) Section 3.2, which is incorporated herein by reference.

[0044] The gNB supports various protocol layers, such as Layer 1 (L1) (ie, physical layer), Layer 2 (L2), and Layer 3 (L3).

[0045] NR’s Layer 2 (L2) is split into the following sublayers: Medium Access Control (MAC), Radio Link Control (RLC), Packet Data Convergence Protocol (PDCP), and Service Data Adaptation Protocol (SDAP), where for example:

[0046] ○The physical layer provides the transport channel to the MAC sublayer;

[0047] ○The MAC sublayer provides logical channels to the RLC sublayer;

[0048] ○ The RLC sublayer provides RLC channels to the PDCP sublayer;

[0049] ○ The PDCP sublayer provides radio bearer to the SDAP sublayer;

[0050] ○The SDAP sublayer provides Quality of Service (QoS) flows to 5GC;

[0051] o Control channels include the Broadcast Control Channel (BCCH) and the Physical Control Channel (PCCH).

[0052] Layer 3 (L3) includes, for example, Radio Resource Control (RRC) in accordance with Section 6 of 3GPP TS 38.300 V16.6.0 (2021-06), which is incorporated herein by reference.

[0053] The gNB Central Unit (gNB-CU) includes, for example, a logical node that hosts, for example, the Radio Resource Control (RRC), Service Data Adaptation Protocol (SDAP) and Packet Data Convergence Protocol (PDCP) protocols of the gNB, or the RRC and PDCP protocols of the en-gNB, which controls the operation of one or more gNB Distributed Units (gNB-DUs). The gNB-CU terminates the F1 interface connected to the gNB-DU. The gNB-CU may also be referred to herein as a CU, a central unit, a centralized unit, or a control unit.

[0054] The gNB distributed unit (gNB-DU) includes logical nodes such as hosting the radio link control (RLC), media access control (MAC) and physical (PHY) layers of, for example, a gNB or en-gNB, and its operation is controlled in part by the gNB-CU. One gNB-DU supports one or more cells. A cell is supported by only one gNB-DU. The gNB-DU terminates the F1 interface connected to the gNB-CU. The gNB-DU may also be referred to as DU or distributed unit in this article.

[0055] As used herein, the term "network node" may refer to any one of a gNB, a gNB-CU or a gNB-DU, or any combination thereof. A RAN (Radio Access Network) node or a network node (such as, for example, a gNB, a gNB-CU or a gNB-DU, or a portion thereof) may be implemented using, for example, a device having at least one processor and / or at least one memory having processor-readable instructions ("program") configured to support and / or supply and / or process CU and / or DU related functions and / or features, and / or at least one protocol (sub) layer of a RAN (Radio Access Network), such as Layer 2 and / or Layer 3. There may be different functional splits between central and distributed units. Examples of such devices and components are described below.

[0056] For example, the gNB-CU and gNB-DU components may be co-located or physically separated. The gNB-DU may be even further split into two parts, for example, one part including a processing device and the other including an antenna. The central unit (CU) may also be referred to as a baseband unit / radio equipment controller / cloud RAN / virtual RAN (BBU / REC / C-RAN / V-RAN), an open RAN (O-RAN) or a portion thereof. The distributed unit (DU) may also be referred to as a remote radio head / remote radio unit / radio equipment / radio unit (RRH / RRU / RE / RU) or a portion thereof. Hereinafter, in various example embodiments of the present disclosure, a network node supporting at least one of a central unit function or a layer 3 protocol of a radio access network may be, for example, a gNB-CU. Similarly, a network node supporting at least one of a distributed unit function or a layer 2 protocol of a radio access network may be, for example, a gNB-DU.

[0057] A gNB-CU may support one or more gNB-DUs. A gNB-DU may support one or more cells and may therefore support a serving cell for a user equipment (UE) or a candidate cell for handover, dual connectivity and / or carrier aggregation.

[0058] User equipment (UE) 150 may be or include a wireless or mobile device, a device having a radio interface for interacting with a RAN (Radio Access Network), a smartphone, a vehicle-mounted device, an IoT device or an M2M device, and other types of user equipment. Such a UE 150 may include: at least one processor; and at least one memory including program code; wherein the at least one memory and the computer program code are configured to cause the device to perform at least certain operations together with the at least one processor, such as, for example, an RRC connection with a RAN. Examples of components of a UE will be combined with Figure 7 Described. In an embodiment, UE 150 may be configured to generate a message (e.g., including a cell ID) to be sent to the RAN via radio (e.g., to reach and communicate with a serving cell). In an embodiment, UE 150 may generate and send and receive an RRC message containing one or more RRCPDUs (packet data units). Those skilled in the art will understand the RRC protocol and other processes that the UE may perform.

[0059] Continue to refer Figure 1In the example of a 5G NR network, the network system 100 provides one or more cells that define a coverage area of ​​the network system 100. As described above, the network system 100 may include a gNB of a 5G NR network, or may include any other device configured to control radio communications and manage radio resources within a cell. As used herein, the term "resource" may refer to radio resources, such as resource blocks (RBs), physical resource blocks (PRBs), radio frames, subframes, time slots, subbands, frequency regions, subcarriers, beams, etc. In an embodiment, the network node 120 may be referred to as a base station.

[0060] Figure 1 An example is provided and is only used to illustrate the network system 100 and the UE 150, which includes multiple devices 160 connected to the UE 150. Those skilled in the art will understand that the network system 100 may include Figure 1 Components not shown in the figure, and it will be understood that other user equipment can communicate with the network system 100. In addition, the UE 150 can communicate with more than two devices 160. In various embodiments, the connection between the UE 150 and the multiple connected devices 160 is wireless. The UE 150 acts as a communication center point for all connected devices 160. The UE 150 is a center point for providing tethered connections and for providing (e.g., audio streaming data and video streaming data) information. For example, when a connection to the UE 150 is not possible, the device 160 is not directly connected to the network system 100.

[0061] Figure 2 yes Figure 1 A 5G NR network may be described as an example of a network system 100, and aspects described below should also be applicable to other types of network systems. The network system may be based on Figure 1 The signals and connections shown are operated so that the UE 150 communicates with the network system 100 through the radio access network 225. In addition, as shown and described herein, the network system can be divided into user plane components and functions and control plane components and functions. Unless otherwise specified, the terms "component", "function" and "service" are used interchangeably herein, and they may refer to instructions executed by one or more processors and implemented by them.

[0062] The following describes example functions of the components. The example functions are illustrative only, and it should be understood that the components described herein may perform other operations and functions. In addition, the connections between components may be virtual connections based on service interfaces, so that any component may communicate with any other component. In this way, any component may act as a service "producer" for any other component (which is a service "consumer") to provide services for network functions.

[0063] For example, a core network 210 is described in the control plane of the network system. The core network 210 may include an authentication server function (AUSF) 211, an access and mobility function (AMF) 212, and a session management function (SMF) 213. The core network 210 may also include a network slice selection function (NSSF) 214, a network exposure function (NEF) 215, a network repository function (NRF) 216, and a unified data management function (UDM) 217, which may include a unified data repository (UDR) 224.

[0064] Additional components and functions of the core network 210 may include an application function (AF) 218, a policy control function (PCF) 219, a network data analysis function (NWDAF) 220, an analysis data repository function (ADRF) 221, a management data analysis function (MDAF) 222, and an operation and management function (OAM) 223.

[0065] The user plane includes UE 150, radio access network (RAN) 225, user plane function (UPF) 226, and data network (DN) 227. RAN 225 may include a combination of Figure 1 The RAN 225 may include one or more components such as one or more network nodes. However, the RAN 225 may not be limited to such components. The UPF 226 provides a connection for data sent through the RAN 225. For example, the DN 226 identifies services from a service provider, Internet access, and third-party services.

[0066] AMF 212 handles connection and mobility tasks. AUSF 211 receives authentication requests from AMF 212 and interacts with UDM 217 to authenticate and verify network responses to determine successful authentication. SMF 213 performs packet data unit (PDU) session management and manages session context with UPF 226.

[0067] NSSF 214 may select a network slice instance (NSI) and determine allowed network slice selection assistance information (NSSAI). This selection and determination is used to set up AMF 212 to provide services to UE 150. NEF 215 ensures third-party access to network services to create specialized network services. NRF 216 acts as a repository for storing network functions to allow these functions to register and discover each other.

[0068] UDM 217 generates authentication vectors for use by AUSF 211 and ADM 212, and provides user identification processing. UDM 217 can be connected to UDR 224, which stores data related to authentication, applications, etc. AF 218 provides application services (e.g., streaming services, etc.) to users. PCF 219 provides policy control functions. For example, PCF 219 can assist in network slicing and mobility management, as well as provide quality of service (QoS) and charging functions.

[0069] NWDAF 220 collects data (e.g., from UE 150 and network systems) to perform network analysis and provide insights into functions that utilize analysis when providing services. ADRF 221 allows consumers to store, retrieve, and delete data and analysis. MDAF 222 provides additional data analysis services for network functions. OAM 223 provides provisioning and management processing functions to manage elements in or connected to the network (e.g., UE 150, network nodes, etc.).

[0070] Figure 2 These are merely examples of components of a network system, and variations are considered to be within the scope of the present disclosure. In an embodiment, a network system may include Figure 2 In an embodiment, the network system may not include other components not shown. Figure 2 In an embodiment, these components and connections can be used with Figure 2 Such and other embodiments are considered to be within the scope of the present disclosure.

[0071] As used herein, the term "signal" may refer to communication signals between UE 150 and device 160 and communication signals between UE 150 and network system 100. In addition, the term "signal" may refer to communication signals between components located in a single device, such as signals between components located in UE 150, or signals between components located in one of devices 160. Those skilled in the art will appreciate that the term "signal" may refer to additional communication signals between components of a single device / apparatus, or between multiple devices / apparatuses, and that the signal may encompass various example transmission technologies, including but not limited to radio frequency (RF) signals, electrical signals, and / or electromagnetic (EM) radiation signals.

[0072] As used herein, the term "data" may refer to information carried in one or more signals. The information may be encoded as digital information or analog information. In various example embodiments, the term "data" may refer to a data stream that may include media data. This may refer to video data (e.g., video stream data) and / or audio data (e.g., audio stream data) and / or other types of data. The terms "audio stream data" and "audio data stream" may be used interchangeably. The terms "video stream data" and "video data stream" may be used interchangeably. Those skilled in the art will appreciate that "data" may refer to additional information that may be carried in a signal. For example, in various embodiments, the term "data" may refer to timing information of a signal and / or synchronization information carried in one or more signals. As used herein, the term "synchronization" may refer to synchronization in time and / or phase, unless otherwise specified.

[0073] As used herein, the term "downlink" may refer to the flow of information from a network system to a UE and then to multiple connected devices. The term "uplink" may refer to the flow of information from multiple connected devices to a UE and then to a network system.

[0074] As used herein, communication with a component of a network system (e.g., a RAN of a network system) may refer to and represent communication with a portion of the RAN, such as communication with a network node (e.g., DU and / or CU), or communication with another portion of the RAN. As used herein, communication with a core network may refer to and represent communication with one or more services / applications of the core network (such as an AMF or another service of the core network).

[0075] As described above and herein, and with continued reference to Figure 1, UE 150 can receive communication signals from network system 100 in downlink, and receive communication signals from any peripheral device (such as device 160 connected thereto) in uplink. The received signal includes data, such as video stream data and / or audio stream data. The data sent (or received) in one signal may be asynchronous in time with the data sent (or received) in another signal. For example, the audio stream data sent (or received) between UE 150 and device 160a may be asynchronous in time with the video stream data sent (or received) between UE 150 and device 160b. In some examples, the lack of synchronization may be caused by processing (queuing, caching) delays and / or transmission delays in device 160, which may cause the signal from device 160a to be received by UE 150 before or after the signal from another device 160b. In addition, although the following description describes various example operations between UE 150 and devices 160a and 160b, these operations are applicable to any connected device 160 (e.g., device 160c).

[0076] For example, according to an example downlink embodiment, UE 150 may receive a signal including both video stream data and audio stream data from network system 100. For example, when UE 150 sends audio stream data to device 160a (e.g., earphone) and sends video stream data to device 160b (e.g., display device), the presentation of each data on the respective devices may be out of sync with each other in time. As a result, the audio played by device 160a and the video displayed by device 160b will be out of sync. This out of sync state may be caused by, for example, the difference in processing and sending / receiving time between device 160a and UE 150 and between device 160b and UE 150, which may cause a time delay in presenting data for one device compared to another device. By synchronizing the presentation of data in each of devices 160a / 160b in time, the quality of experience (QoE) of the end user of the device may be improved.

[0077] In the example case where UE 150 may be receiving a signal including audio stream data from device 160a and a signal including video stream data from device 160b, the data in the two signals may not be synchronized. Therefore, the resultant transmission of the data in the signal from UE 150 to another entity (such as network system 100) in the uplink direction may result in an unsynchronized presentation. Synchronizing the audio stream data and the video stream data into the signal sent by UE 150 to network system 100 may improve the QoE of the data contained in the display signal for the user.

[0078] Thus, example techniques for synchronizing data in time for presentation are described herein.In an example embodiment, synchronization includes synchronizing data in signals received by UE 150 from network system 100 for transmission to device 160, which may be referred to as a "downlink" transmission direction, as described above.

[0079] In another example embodiment, synchronization includes receiving data from device 160a, which in one example may be referred to as "first data"; and receiving data from device 160b, which in one example may be referred to as "second data". UE 150 may generate a resultant signal including the first data and the second data that are synchronized to be sent to network system 100. As described above, this example technique may be referred to as an "uplink" transmission direction.

[0080] Figure 3A 1 is a diagram of an example embodiment of a UE 150, a connected device 160, and a network system 100 showing example downlink communications according to one illustrative aspect of the present disclosure. Figure 3A As shown, UE 150 includes a media access function (MAF) 151. MAF 151 in UE 150 provides processing for media received and / or sent by UE 150. Figure 3A As shown, MAF 151 includes at least an audio decoder and processing function 152, a video decoder and processing function 153, a delay estimation function 154, and an audio / video synchronization unit function 155. In an exemplary embodiment, the delay estimation function 154 and the audio / video synchronization unit function 155 may be located in or on a submodule within MAF 151. Those skilled in the art will appreciate that other functions may be included in MAF 151. The audio decoder and processing function 152, the video decoder and processing function 153, the delay estimation function 154, and the audio / video synchronization unit function 155 may be operatively coupled and communicate with each other to share signals and information with each other.

[0081] The audio decoder and processing function 152 performs decoding of the audio data stream in the signal received by the UE 150. In some embodiments, the audio data in the audio data stream may be compressed to facilitate the transmission of the data. Therefore, the audio decoder and processing function 152 may decompress the audio data in the signal received by the UE 150.

[0082] Similarly, the video decoder and processing function 153 performs decoding of the video data stream in the signal received by the UE 150. In some embodiments, the video data in the video data stream may be compressed to facilitate the transmission of the data. Therefore, the video decoder and processing function 153 may decompress the video data in the signal received by the UE 150.

[0083] exist Figure 3A In the embodiment, the delay estimation function 154 performs an estimation of one or more downlink delays between the UE 150 and the devices 160 connected to the UE 150. In various embodiments, the delay estimation may include an estimation of the processing delay in each device 160 (such as devices 160a and 160b) and the transmission delay to each device 160. This delay may reflect the time delay between the UE 150 sending the data and the device (160a or 160b) presenting the data to the user. In an example embodiment, the delay estimation may include the use of round-trip time (RTT) delay estimation techniques, which are familiar to those skilled in the art. In some examples, the RTT delay estimation may include the use of the following: for example, the Internet Control Message Protocol (ICMP) protocol using the RFC792 message format, and / or the Real-time Transport Protocol / Real-time Control Protocol (RTP / RTCP) using the RFC3550 message format. The delay estimation function 154 of the MAF 151 estimates the delay allowing the MAF 151 to determine the communication delay between the UE 150 and each device 160. Those skilled in the art will appreciate that additional delay measurements may be utilized and additional techniques for estimating delay may be employed in the delay estimation function 154 .

[0084] The audio / video synchronization unit function 155 configures the data transmission to the device 160 so that the data presentation of the device 160 is synchronized in time and / or phase or approximately synchronized in time and / or phase. In various embodiments, for example, the audio / video synchronization unit function 155 can synchronize the audio stream data (e.g., first data) in the signal for transmission to the device 160a with the video stream data (e.g., second data) in the signal for transmission to the device 160b in time. In some embodiments, the audio / video synchronization unit function 155 can utilize the delay of each device 160 determined by the delay estimation function 154 to synchronize the presentation of the data to the user of the device 160a with the presentation of the data to the user of the device 160b in time. In some embodiments, the users of the devices 160a and 160b can be the same users. Therefore, the audio / video synchronization unit function 155 enables the users of the devices 160a and 160b to perceive the presentation of the data by the two devices as synchronized presentation.

[0085] The audio / video synchronization unit function 150 is only an example of a synchronization function. Generally, the synchronization function is used to configure the downlink transmission so that the device presents data in a time synchronization or approximate time synchronization mode. In various embodiments, due to factors such as the transmission delay to the device 160 that changes and / or the processing requirements at the device 160 that change, the position of the device 160 that changes, the synchronization unit may not be able to perfectly present synchronously in time. However, the synchronization function can be able to present approximately synchronous data so that one or more users still perceive that the presentation is synchronous in time. Therefore, as used in this article, the term "approximate synchronization" (and its variants) refers to synchronization that may not be perfect but can still be perceived by one or more users as being synchronous in time. The following description will continue to be described with audio and video synchronization as an example. However, it is intended to regard any description of audio and video synchronization as referring to the synchronization of any data type. In an exemplary embodiment, IEEE 1588-2002, IEEE 1588-2008, and IEEE 1588-2019 types of protocols can be used within the synchronization function.

[0086] In addition, if Figure 3A As shown, in some embodiments, device 160a can be a set of wireless headphones and can also include a device for receiving audio input data (e.g., a microphone). In some embodiments, device 160b can include a wireless device for displaying video data (e.g., AR glasses), and in some implementations, can also include a device for capturing images (e.g., a camera). Both devices 160 can include an AR / MR application 161 for processing data (e.g., encoding / decoding) and a wireless communication function 162 for facilitating the transmission and reception of wireless signals, and are used for the tethering process at the same time. Those skilled in the art will understand that the components shown in device 160 are merely examples, and other components and functions may be present in device 160.

[0087] As mentioned above, Figure 3A The example operations shown may be referred to as the downlink direction. Figure 3A As shown, UE 150 can communicate with network system 100 and can receive signals (e.g., signals including acquired content) from network system 100. In some embodiments, the acquired content may include streaming data, such as live streaming data or on-demand data. In other example embodiments, the acquired content may be downlink information received during a video conference and / or an audio conference. In some examples, the acquired content may include video streaming data and audio streaming data.

[0088] After UE 150 receives the acquired content from network system 100, audio decoder and processing function 152 performs decoding and processing of audio stream data in the acquired content. In some embodiments, audio decoder and processing function 152 can extract audio stream data from the acquired content for decoding / processing. Additionally, in some examples, video decoder and processing function 153 performs decoding and processing of video stream data in the acquired content. In some embodiments, video decoder and processing function 153 can extract video stream data from the acquired content for decoding / processing.

[0089] In some examples, the audio / video synchronization unit function 155 receives decoded / processed audio stream data from the audio decoder and processing function 152 and receives decoded / processed video stream data from the video decoder and processing function 153 using information related to the delays of the devices 160a and 160b determined by the delay estimation function 154. The audio / video synchronization unit function 155 generates a signal including the audio stream data for transmission to the device 160a based on the determined delay of the device 160a, and generates a signal including the video stream data for transmission to the device 160b based on the delay of the device 160b. The delays in various embodiments may include transmission delays and / or processing delays. For example, the delays may include transmission delays for propagating the signal to the respective device 160 and / or may also include delays caused by processing delays in the respective device 160.

[0090] In some embodiments, the transmission of a signal including audio stream data (audio data out / downlink signal) to device 160a may be sent at a first time, and the transmission of a signal including video stream data (video data out / downlink signal) to device 160b may be sent at a second time. For example, the transmission of the audio data out / downlink signal may be sent after a time delay. In some examples, the delay may be encoded in the audio data out / downlink signal and the video data out / downlink signal. For example, one or both of the audio data out / downlink signal and the video data out / downlink signal may include data that provides information to indicate to the respective device 160 the time for presenting the data. For example, the video data out / downlink signal may include an indication of time = 0 (t 0 ), and the audio data output / downlink signal may include information indicating time = 1 (t 1 ) at the start time. In this case, device 160b can 0 Rendering (eg, rendering video data), while device 160a at time t 1Render its data (eg, play audio). The values ​​t0 and t1 are delays, which may include transmission delays for the signal to propagate to the respective device 160 , and / or may also include delays due to processing delays in the respective device 160 .

[0091] therefore, Figure 3A The illustrated example downlink operation provides for splitting a signal received by UE 150 from network system 100 into two component signals (e.g., audio stream data and video stream data signals). The two signals are synchronized in time by audio / video synchronization unit function 155, and synchronized data output is provided for ultimate synchronized presentation and / or perception by one or more users, such as synchronized audio and video rendering and / or reproduction.

[0092] The following describes the process from the perspective of UE 150. Figure 3A Various operations. The operations include receiving, by a UE, a first signal (e.g., acquired content) from a first device (e.g., network system 100), the first signal including first data (e.g., audio stream data) for presentation by a second device (e.g., device 160a) communicating with the UE and second data (e.g., video stream data) for presentation by a third device (e.g., device 160b) communicating with the UE, the first data and the second data being synchronized with each other in time; extracting, by the UE, the first data from the first signal; extracting, by the UE, the second data from the first signal; determining a timing for sending a second signal (e.g., audio data output / downlink) including the first data to the second device and a third signal (e.g., video data output / downlink) including the second data to the third device, wherein the timing is configured to approximately synchronize presentation of the first data by the second device and presentation of the second data by the third device in time; sending a second signal to the second device based on the timing; and sending a third signal to the third device based on the timing.

[0093] The following describes the process from the perspective of UE 150. Figure 3AVarious additional operations. The operations include receiving, by the UE, a first signal (e.g., acquired content) from a first device (e.g., network system 100), the first signal including first data (e.g., audio stream data) for presentation by a second device (e.g., device 160a) communicating with the UE and second data (e.g., video stream data) for presentation by a third device (e.g., device 160b) communicating with the UE, the first data and the second data being synchronized with each other in time; extracting, by the UE, the first data from the first signal; extracting, by the UE, the second data from the first signal; determining, by the UE, timing for presentation of the first data by the second device and presentation of the second data by the third device, the timing being configured to approximately synchronize presentation of the first data by the second device and presentation of the second data by the third device in time; sending, by the UE, a second signal (e.g., audio data output / downlink) including the first data to the second device; and sending, by the UE, a third signal (e.g., video data output / downlink) including the second data to the third device, wherein at least one of the second signal or the third signal includes a timing-based encoding presentation delay.

[0094] In some example embodiments, the signal may be received by UE 150 from device 160 for transmission to network system 100 . Figure 3B is a diagram of an example embodiment of a UE 150, a connected device 160, and a network system 100 showing example uplink communications according to one illustrative aspect of the present disclosure. Figure 3A , the UE 150 includes a media access function (MAF) 151. The MAF 151 in the UE 150 provides processing for media received and / or transmitted by the UE 150. Figure 3B The MAF 151 shown includes at least a delay estimation function 154, an audio / video mixer unit function 156, an audio encoder and processing function 157, and a video encoder and processing function 158. In an example embodiment, the delay estimation function 154 and the audio / video mixer unit function 156 may be located in or on a submodule within the MAF 151, and in some embodiments, may be located on a submodule of the MAF 151. Figure 3A The MAF 151 is shown in the same submodule as the audio / video synchronization unit function 155. Those skilled in the art will appreciate that other functions may be included in the MAF 151. The delay estimation function 154, the audio / video mixing unit function 156, the audio encoder and processing function 157, and the video encoder and processing function 158 may be operatively coupled and communicate with each other to share signals and information with each other.

[0095] The audio encoder and processing function 157 performs encoding of the audio data stream in the signal received by the UE 150. In some embodiments, the audio data in the audio data stream can be compressed to facilitate the transmission of the data. Therefore, the audio encoder and processing function 157 can compress the audio data in the signal received by the UE 150 from the device 160a.

[0096] Similarly, the video encoder and processing function 158 performs encoding of the video data stream in the signal received by the UE 150. In some embodiments, the video data in the video data stream can be compressed to facilitate the transmission of the data. Therefore, the video encoder and processing function 158 can compress the video data in the signal received by the UE 150 from the device 160b.

[0097] Similar to the delay estimation function 154, the delay estimation function 154 performs an estimation of one or more delays between the UE 150 and the device 160 connected to the UE 150. In various embodiments, the delay estimation may be similar to the above-described Figure 3A Those skilled in the art will appreciate that additional delay measurements may be utilized and additional techniques for estimating delay may be employed in the delay estimation function 154.

[0098] The audio / video mixer unit function 156 synchronizes the data received by the UE 150 from the devices 160 in time for transmission in a signal to the network system 100. In various embodiments, for example, the audio / video synchronization unit function 156 may synchronize the audio stream data (e.g., first data) received from the device 160a with the video stream data (e.g., second data) received from the device 160b in time for transmission in a signal to the network system 100. In some embodiments, the audio / video synchronization mixer function 156 may utilize the delay of each device 160 determined by the delay estimation function 154 so as to synchronize the first data and the second data in a common signal to be sent to the network system 100. That is, the audio / video mixer unit function 156 may mix, for example, the audio data and the video data into a single audio / video signal. In an exemplary embodiment, the audio data stream and the video data stream may be identified using header information used in the Real-time Transport Protocol (RTP) IETF RFC 3550 or the Real-time Streaming Protocol (RTSP) IETF RFC 2326, through which the audio signal and the video signal are sent by different channels, and their corresponding header information has identifiers, as will be understood by those skilled in the art.

[0099] In some examples, audio data is transmitted more frequently and data packets may be relatively small, while video data is transmitted less frequently and data packets may be larger than audio packets. Therefore, in some embodiments, the audio / video mixer unit function 156 may generate a signal for transmission by mixing packets according to received timestamp information, and synchronizing the audio data packets with corresponding video data packets in time based on delay information received from the delay estimation function 154. Therefore, the audio / video mixer unit function 156 creates synchronized audio stream / video stream data for transmission to the network system 100.

[0100] Devices 160a and 160b may be similar to those described above. Figure 3A Devices 160a and 160b described in .

[0101] As mentioned above, Figure 3B The example operations shown may be referred to as the uplink direction. Figure 3B As shown, UE 150 can communicate with device 160a and can receive an audio data signal (e.g., an audio data input / uplink signal) containing audio stream data. Additionally, UE 150 can communicate with device 160b and can receive a video data signal (e.g., a video data input / uplink signal) containing video stream data. The audio data in the audio data signal can be, for example, audio data captured by a microphone of device 160a, and the video data in the video data signal can be captured by, for example, an image capture device of device 160b.

[0102] After UE 150 receives the audio data signal from device 160a, audio encoder and processing functionality 157 performs encoding and processing of the audio stream data. Additionally, in some examples, video encoder and processing functionality 158 performs encoding and processing of video stream data.

[0103] In some examples, the audio / video synchronization mixer function 156 receives the encoded / processed audio stream data from the audio encoder and processing function 157 and receives the encoded / processed video stream data from the video encoder and processing function 158 using the information related to the delay of the devices 160a and 160b determined by the delay estimation function 154. The audio / video mixer unit function 156 generates a signal including both the audio stream data and the video stream data synchronized for transmission to the network system 100 based on the determined delay of the device 160a and the delay of the device 160b.

[0104] The delay in various embodiments may include transmission delay and / or processing delay. For example, the delay may include transmission delay for propagation of the signal to the corresponding device 160, and / or may also include delay caused by processing delay in the corresponding device 160. For example, the audio stream data at time = 0 (t 0 ) is received from device 160a and the corresponding video stream data is received at time = 1 (t 1 ) is received from device 160b, the audio / video synchronization mixer function 156 can synchronize both the audio stream data and the video stream data to start at the same time in the combined signal to be sent to the network system 100. The audio / video synchronization mixer function is just one example. In general, the mixer function can mix multiple data of any type and synchronize them in time.

[0105] therefore, Figure 3B The example uplink operation shown provides for combining signals received by UE 150 from devices 160a and 160b at different times into one signal as synchronized data for transmission to network system 100 .

[0106] The following describes the process from the perspective of UE 150. Figure 3B Various operations. The operations include receiving, by the UE, a first signal (e.g., voice data input / uplink) including first data (e.g., audio stream data) from a first device (e.g., device 160) communicating with the UE; receiving, by the UE, a second signal (e.g., image data input / uplink) including second data (e.g., video stream data) from a second device (e.g., device 160b) communicating with the UE; determining, by the UE, a timing that synchronizes the first data and the second data in time; generating, by the UE, a third signal (e.g., to a network) including the first data and the second data based on the timing, the first data and the second data being synchronized in time in the third signal; and sending a third signal to a third device (e.g., network system 100).

[0107] Figure 4 is a diagram of an example embodiment of a UE 150 , a connected device 160 , and a network system 100 , showing combined downlink and uplink communications according to one illustrative aspect of the present disclosure. Figure 4 The components and operations shown are similar to those described above in Figure 3A and Figure 3B 1, but shows the various components combined in the MAF 151.

[0108] Figure 5A and Figure 5Bis a diagram of example signals and operations of a network system according to one illustrative aspect of the present disclosure. The following paragraphs will describe various signals and operations. It should be understood that the described signals may have associated operations, and the described operations may have associated signals. Figure 5A and Figure 5B The example signals and operations shown depict example downlink direction communications as described above.

[0109] In operation 500, the initial content may be acquired by the UE 150 from the network system 100. For example, in operation 500, the UE 150 may receive a signal including audio stream data and video stream data via the wireless communication function 159, for example.

[0110] At operation 501a, a tether link between UE 150 and device 160a is initialized. At operation 501b, a tether link between UE 150 and device 160b is initialized. In some examples, operations 501a and 501b may be performed asynchronously, for example, the operations are not coordinated in time.

[0111] At operation 501c, UE 150 sends a ping request to device 160b, and device 160b receives the ping request. In response, at operation 501d, device 160b sends a ping response to UE 150, and UE 150 receives the ping response. At operation 501e, device 160b sends a delay report to UE 150, and UE 150 receives the delay report. The delay report may include information about processing delays, such as queuing and buffering delays, and other possible processing delays.

[0112] At operation 501f, UE 150 sends a ping request to device 160a, and device 160a receives the ping request. In response, at operation 501g, device 160a sends a ping response to UE 150, and UE 150 receives the ping response. At operation 501h, device 160a sends a delay report to UE 150, and UE 150 receives the delay report. The delay report may include information about processing delays, such as queuing and buffering delays, and other possible processing delays.

[0113] In operation 502a, a tethered link is established between UE 150 and device 160b, and the link may include an estimated delay report, which estimates one or more delays between UE 150 and device 160b, such as transmission delays and processor delays described herein. In operation 502b, a tethered link is established between UE 150 and device 160a, and the link may include an estimated delay report, which estimates one or more delays between UE 150 and device 160a, such as transmission delays and processor delays described herein. In some examples, operations 502a and 502b may be performed asynchronously, for example, these operations are not coordinated in time and / or phase. In general, operations 501a-502b may cover tethering and delay calculation / estimation of several peer devices (e.g., 160a / 160b) connected in parallel to UE150.

[0114] In addition, with respect to operations 503a-507d, as explained in more detail below, after a tether is established between device 160a / 160b and UE 150, operations associated with signals or messages received by UE 150 from device 160a or 160b may be asynchronous (e.g., not coordinated in time), while operations associated with signals or messages sent from UE 150 to devices 160a and 160b may be synchronized with each other (e.g., coordinated in time).

[0115] Referring now to operations 503a-505d, these operations may collectively encompass Real Time Streaming Protocol (RTSP) establishment of several peer devices (e.g., 160a / 160b) connected in parallel to UE 150. In various example embodiments, RTSP performs establishment and control of a media session between a sending device of media (e.g., video stream data / audio stream data) and a device receiving the media.

[0116] At operation 503a, a request for a video sequence message is sent from device 160b, and UE 150 receives the request for the video sequence message. At operation 503b, a request for an audio sequence message is sent from device 160a, and UE 150 receives the request for the audio sequence message. At operation 503c, UE 150 responds to the request for the video sequence message with a video available message received by device 160b. At operation 503d, UE 150 responds to the request for the audio sequence message with an audio available message received by device 160a.

[0117] At operation 504a, a request (ask) for a video description message is sent from device 160b, and UE 150 receives the request for the video description message. At operation 504b, a request (ask) for an audio description message is sent from device 160a, and UE 150 receives the request for an audio sequence message. At operation 504c, UE 150 responds to the request for the video description message with a video available message received by device 160b. At operation 504d, UE 150 responds to the request for the audio description message with an audio available message received by device 160a.

[0118] At operation 505a, a setup request for a video track message is sent from device 160b, and UE 150 receives the setup request for the video track message. At operation 505b, a setup request for an audio track message is sent from device 160a, and UE 150 receives a request message for an audio track. At operation 505c, UE 150 responds to the setup request for the video track message with a video available message received by device 160b. At operation 505d, UE 150 responds to the setup request for the audio track message with an audio available message received by device 160a.

[0119] After the RTSP connection is established, operations 506a-507d can be directed to use the above-mentioned RTP protocol and / or user data protocol (UDP) to transmit synchronous audio and video data. For example, in operation 506a, device 160b sends a request video data message to UE 150, and UE 150 receives the request video data message. In operation 506b, device 160a sends a request audio data message to UE 150, and UE 150 receives the request audio data message. In response to UE 150 receiving the request video data message, UE 150 sends a "ready to receive" message to device 160b in operation 506c, and device 160b receives the "ready to receive" message. In response to receiving the request audio data message, UE 150 sends a "ready to receive" message to device 160a in operation 506c, and device 160a receives the "ready to receive" message.

[0120] Once the UE 150 sends a "ready to receive" message to the respective devices 160 (e.g., at operations 506c and 506d), the UE 150 sends the video and audio data to these devices. In various embodiments, the transmitted video and audio data may be based on the above-described techniques (such as Figure 3AFor example, in operation 507a, UE 150 sends a first video data packet to device 160b, and device 160b receives the first data packet. In operation 507b, UE 150 sends a first audio data packet to device 160a, and device 160a receives the first audio packet. Figure 5A and Figure 5B 160b, and device 160b continues to receive subsequent video packets. UE 150 also continues to send subsequent audio packets to device 160a, and device 160a continues to receive subsequent audio packets. The transmission of subsequent video packets and audio packets continues until UE 150 sends the last video packet and the last audio packet, as shown in operations 507c-507d.

[0121] In operation 507c, the UE 150 transmits the last video data packet to the device 160b, and the device 160b receives the last data packet. In operation 507d, the UE 150 transmits the last audio data packet to the device 160a, and the device 160a receives the last video packet.

[0122] After device 160b receives the last video data packet and device 160a receives the last audio data packet, at operations 508a-508d, the connection of audio and video data between device 160 and UE 150 may be terminated. In various example embodiments, the operations and signals at operations 508a-508d may be asynchronous.

[0123] At operation 508a, device 160b sends a "request to end video session" message to UE 150, and UE 150 receives the "request to end video session" message. At operation 508b, device 160a sends a "request to end audio session" message to UE 150, and UE 150 receives the "request to end audio session" message. In response to receiving the "request to end video session" message, at operation 508c, UE 150 sends a "confirmation to end video session" message to device 160b, and device 160b receives the "confirmation to end video session" message. In response to receiving the "request to end audio session" message, at operation 508d, UE 150 sends a "confirmation to end audio session" message to device 160a, and device 160a receives the "confirmation to end audio session" message.

[0124] Figure 5A and Figure 5B The signals and operations are illustrative only, and variations are considered to be within the scope of the present disclosure. In an embodiment, the signals and operations may include Figure 5A or Figure 5BOther signals and operations not shown in the embodiment. In an embodiment, the signals and operations may not include Figure 5A and Figure 5B In an embodiment, these signals and operations can be used in conjunction with Figure 5A and Figure 5B Such and other embodiments are considered to be within the scope of the present disclosure.

[0125] Combined with Figure 3A as well as Figure 5A and Figure 5B In the manner described, the UE schedules the transmission of synchronization data to the devices connected to the UE so that the presentation of the data by the devices is synchronized in time and / or phase or approximately synchronized in time and / or phase. Figure 3B and Fig. 6A and Figure 6B In the manner described in the disclosure, a UE receives multiple asynchronous data communications from multiple connected devices and sends a communication signal including synchronous data received from the multiple connected devices.

[0126] Fig. 6A and Figure 6B is a diagram of example signals and operations of a network system according to one illustrative aspect of the present disclosure. The following paragraphs will describe various signals and operations. It should be understood that the described signals may have associated operations, and the described operations may have associated signals. Fig. 6A and Figure 6B The example signals and operations shown depict example uplink direction communications as described above.

[0127] In operation 600, initial content may be acquired by UE 150 from network system 100. For example, in operation 600, UE 150 may receive a signal including audio stream data and video stream data, or a request for audio stream and video stream data, via wireless communication function 159, for example.

[0128] At operation 601a, a tether link between UE 150 and device 160a is initialized. At operation 601b, a tether link between UE 150 and device 160b is initialized. In some examples, operations 601a and 601b may be performed asynchronously, for example, the operations are not coordinated in time.

[0129] At operation 601c, UE 150 sends a ping request to device 160b, and device 160b receives the ping request. In response, at operation 601d, device 160b sends a ping response to UE 150, and UE 150 receives the ping response. At operation 601e, device 160b sends a delay report to UE 150, and UE 150 receives the delay report. The delay report may include information about processing delays, such as queuing and buffering delays, and other possible processing delays.

[0130] At operation 601f, UE 150 sends a ping request to device 160a, and device 160a receives the ping request. In response, at operation 601g, device 160a sends a ping response to UE 150, and UE 150 receives the ping response. At operation 601h, device 160a sends a delay report to UE 150, and UE 150 receives the delay report. The delay report may include information about processing delays, such as queuing and buffering delays, and other possible processing delays.

[0131] In operation 602a, a tethered link is established between UE 150 and device 160b, and the link may include an estimated delay report, which estimates one or more delays between UE 150 and device 160b, such as transmission delays and processor delays described herein. In operation 602b, a tethered link is established between UE 150 and device 160a, and the link may include an estimated delay report, which estimates one or more delays between UE 150 and device 160a, such as transmission delays and processor delays described herein. In some examples, operations 602a and 602b may be performed asynchronously, for example, these operations are not coordinated in time and / or phase. In general, operations 601a-602b may cover tethering and delay calculation / estimation of several peer devices (e.g., 160a / 160b) connected in parallel to UE150.

[0132] In addition, with respect to operations 603a-605d, as explained in more detail below, after a tether is established between device 160a / 160b and UE 150, operations associated with signals or messages received by UE 150 from device 160a or 160b may be asynchronous (e.g., not coordinated in time and / or phase), while operations associated with signals or information sent from UE 150 to network system 100 may be synchronized with each other (e.g., coordinated in time and / or phase).

[0133] Referring now to operations 603a-603d, these operations may collectively encompass Real Time Streaming Protocol (RTSP) establishment for several peer devices (e.g., 160a / 160b) connected in parallel to UE 150. In various example embodiments, RTSP performs establishment and control of a media session between a sending device of media (e.g., video stream data / audio stream data) and a device receiving the media.

[0134] In operation 603a, the request to send uplink video data stream sequence message is sent from equipment 160b, and UE 150 receives the request to send uplink video data stream sequence message. In operation 603b, the request to send uplink audio data stream sequence message is sent from equipment 160a, and UE 150 receives the request to send uplink audio data stream sequence message. In operation 603c, UE 150 answers the request to send uplink video data stream sequence message with the sending video data stream message received by equipment 160b. In operation 603d, UE 150 answers the request to send uplink audio data stream sequence message with the sending audio data stream message received by equipment 160a.

[0135] Operations 604a-605b may refer to synchronous audio and video data transmission using the RTP protocol described above and / or UDP of several peers in parallel. In various embodiments, the transmitted video and audio data may be transmitted according to the techniques described above (e.g., Figure 3B as shown) to send.

[0136] For example, in operation 604a, device 160b sends a first video data packet to UE 150, and UE 150 receives the first video data packet. In operation 604b, device 160a sends a first audio data packet to UE 150, and UE 150 receives the first audio packet. In operation 604c, UE 150 sends a first video data packet to network system 100, and network system 100 receives the first video data packet. In operation 604d, UE 150 sends a first audio data packet to network system 100, and network system 100 receives the first audio data packet. Operations 604c and 604d may be sent synchronously in time and / or phase, or approximately synchronously in time and / or phase.

[0137] although Fig. 6A and Figure 6B160a, and UE 150 continues to receive subsequent video data packets from device 160b and subsequent audio data packets from device 160a, and UE 150 continues to receive subsequent video and audio data packets from respective devices 160b and 160a. Then, UE 150 continues to send subsequent video data packets and subsequent audio data packets to network system 100, and network system continues to receive subsequent video and audio data packets. As shown in operations 605a-605d, the reception and transmission of subsequent video data packets and audio data packets continue until UE 150 receives and sends the last video data packet and the last audio data packet.

[0138] In operation 605a, device 160b sends the last video data packet to UE 150, and UE 150 receives the last video data packet. In operation 605b, device 160a sends the last audio data packet to UE 150, and UE 150 receives the last audio packet. In operation 605c, UE 150 sends the last video data packet to network system 100, and network system 100 receives the last video data packet. In operation 605d, UE 150 sends the last audio data packet to network system 100, and network system 100 receives the last audio data packet. Operations 605c and 605d may be sent synchronously in time and / or phase, or approximately synchronously in time and / or phase.

[0139] After the network system 100 receives the last video data packet and the last audio data packet, the connection of audio and video data between the device 160 and the UE 150 may be terminated at operations 606a-606d. In various example embodiments, the operations and signals at operations 606a-606d may be asynchronous.

[0140] At operation 606a, device 160b sends a "request to end video session" message to UE 150, and UE 150 receives the "request to end video session" message. At operation 606b, device 160a sends a "request to end audio session" message to UE 150, and UE 150 receives the "request to end audio session" message. In response to receiving the "request to end video session" message, at operation 606c, UE 150 sends a "confirm end video session" message to device 160b, and device 160b receives the "confirm end video session" message and device 160b receives the "confirm end video session" message. In response to receiving the "request to end audio session" message, at operation 606b, UE 150 sends a "confirm end audio session" message to device 160a, and device 160a receives the "confirm end audio session" message.

[0141] Fig. 6Aand Figure 6B The signals and operations are illustrative only, and variations are considered to be within the scope of the present disclosure. In an embodiment, the signals and operations may include Fig. 6A and Figure 6B Other signals and operations not shown in the figure. In an embodiment, the signals and operations may not include Fig. 6A and Figure 6B In an embodiment, these signals and operations can be used in conjunction with Fig. 6A and Figure 6B Such and other embodiments are considered to be within the scope of the present disclosure.

[0142] Reference now Figure 7 , a block diagram of example components of a network device of a UE, a connected device, or (e.g., a RAN or core network) is shown. The device includes an electronic storage device 710, a processor 720, a network interface 740, and a memory 750. The various components can be communicatively coupled to each other. The processor 720 can be and can include any type of processor, such as a single-core central processing unit (CPU), a multi-core CPU, a microprocessor, a digital signal processor (DSP), a system on a chip (SoC), or any other type of processor. The memory 750 can be a volatile type of memory, such as RAM, or a non-volatile type of memory, such as NAND flash memory. The memory 750 includes processor-readable instructions that are executable by the processor 720 to cause the device to perform various operations, including the operations described herein, such as in conjunction with Figure 3A-Figure 6B The operations are shown and described.

[0143] Electronic storage 710 may be and include any type of electronic storage for storing data, such as a hard drive, a solid state drive, and / or an optical disk, among other types of electronic storage. Electronic storage 710 stores processor-readable instructions for causing the device to perform its operations, and stores data associated with such operations, such as storing data related to the 5G NR standard and other data. Network interface 740 may implement wireless network technologies, such as 5G NR and / or other wireless network technologies.

[0144] Figure 7 The components shown are examples only, and those skilled in the art will appreciate that the device includes other components not shown, and may include multiples of any of the components shown. Such and other embodiments are considered to be within the scope of the present disclosure.

[0145] Additional embodiments of the present disclosure include the following examples.

[0146] Example 1.1 A user equipment (UE), comprising:

[0147] at least one processor; and

[0148] At least one memory storing instructions that, when executed by at least one processor, cause the user equipment to at least perform:

[0149] receiving a first signal from a first device, the first signal comprising first data for presentation by a second device in communication with the UE and second data for presentation by a third device in communication with the UE, the first data and the second data being synchronized in time with each other;

[0150] extracting first data from the first signal;

[0151] extracting second data from the first signal;

[0152] determining a timing for sending a second signal including the first data to the second device and a third signal including the second data to the third device, the timing being configured to approximately synchronize presentation of the first data by the second device and presentation of the second data by the third device in time;

[0153] sending a second signal to a second device based on the timing; and

[0154] A third signal is sent to a third device based on the timing.

[0155] Example 1.2. The user equipment of Example 1.1, wherein determining the timing comprises determining processing and transmission delays of the second device.

[0156] Example 1.3. The user equipment of Example 1.1 or 1.2, wherein determining the timing comprises determining processing and transmission delays of the third device.

[0157] Example 1.4. The user equipment of any of Examples 1.1 to 1.3, wherein sending the second signal to the second device is at a first time, and sending the third signal to the third device is at a second time.

[0158] Example 1.5. The user equipment of any of Examples 1.1 to 1.4, wherein sending the second signal to the second apparatus precedes sending the third signal to the third apparatus.

[0159] Example 1.6. A user device according to any one of Examples 1.1 to 1.3, wherein the instructions, when executed by at least one processor, also cause the user device to perform encoding of a first presentation time delay in a second signal, the first presentation time delay being configured to cause presentation of the first data to be approximately synchronized in time with presentation of the second data in a third signal.

[0160] Example 1.7. The user equipment of any of Examples 1.1 to 1.6, wherein the first data comprises a first data stream and the second data comprises a second data stream.

[0161] Example 1.8. A user device according to Examples 1.1 to 1.7, wherein the first data stream is an audio data stream and the second data is a video data stream.

[0162] Example 1.9. A user device according to any of Examples 1.1 to 1.8, wherein the instructions, when executed by at least one processor, further cause the user device to perform decoding and processing of the first data.

[0163] Example 1.10. A user device according to any of Examples 1.1 to 1.9, wherein the instructions, when executed by the at least one processor, further cause the user device to perform decoding and processing of the second data.

[0164] Example 2.1. A user equipment (UE), comprising:

[0165] at least one processor; and

[0166] at least one memory storing instructions that, when executed by at least one processor, cause the user equipment to at least:

[0167] receiving a first signal including first data from a first device in communication with the UE;

[0168] receiving a second signal including second data from a second device in communication with the UE;

[0169] determining a timing for synchronizing the first data with the second data in time;

[0170] Based on the timing, generating a third signal including the first data and the second data, the first data and the second data being synchronized in time in the third signal; and

[0171] A third signal is sent to a third device.

[0172] Example 2.2. The user equipment of Example 2.1, wherein determining the timing comprises determining processing and transmission delays of the first device.

[0173] Example 2.3. The user equipment of Example 2.1 or 2.2, wherein determining the timing comprises determining processing and transmission delays of the second device.

[0174] Example 2.4. The user equipment of Example 2.3, wherein generating the third signal comprises mixing first data of the first signal with second data of the second signal into the third signal.

[0175] Example 2.5. The user equipment of any of Examples 2.1 to 2.4, wherein generating the third signal comprises synchronizing a presentation time of the first data with a presentation time of the second data in the third signal for presentation by a third device.

[0176] Example 2.6. The user equipment of Example 2.5, wherein synchronizing a presentation time of the first data with a presentation time of the second data in a third signal for presentation by a third device comprises synchronizing data packets of the first data with data packets of the second data in time.

[0177] Example 2.7. The user equipment of any of Examples 2.1 to 2.6, wherein the first data comprises a first data stream and the second data comprises a second data stream.

[0178] Example 2.8. The user device of Example 2.7, wherein the first data stream is an audio data stream and the second data is a video data stream.

[0179] Example 2.9. A user device according to any of Examples 2.1 to 2.8, wherein the instructions, when executed by at least one processor, further cause the user device to perform encoding and processing of the first data.

[0180] Example 2.10. A user device according to any of Examples 2.1 to 2.9, wherein the instructions, when executed by at least one processor, further cause the user device to perform encoding and processing of the second data.

[0181] Example 3.1. A method comprising:

[0182] receiving, by a user equipment (UE), a first signal from a first device, the first signal comprising first data for presentation by a second device in communication with the UE and second data for presentation by a third device in communication with the UE, the first data and the second data being synchronized in time with each other;

[0183] Extracting, by the UE, first data from the first signal;

[0184] Extracting, by the UE, second data from the first signal;

[0185] Determining, by the UE, timing for presentation of the first data by the second device and presentation of the second data by the third device, the timing being configured to approximately synchronize presentation of the first data by the second device and presentation of the second data by the third device in time;

[0186] Sending, by the UE, a second signal including the first data to the second device; and

[0187] The UE sends a third signal including the second data to the third device,

[0188] Wherein at least one of the second signal or the third signal comprises a timing-based encoded presentation delay.

[0189] Example 3.2. The method of Example 3.1, wherein the second signal and the third signal are sent sequentially.

[0190] Example 3.3. The method according to any one of Examples 3.1 to 3.2, further comprising:

[0191] determining a first transmission delay from the UE to the second device; and

[0192] determining a second transmission delay from the UE to the third device,

[0193] Wherein the timing is based on a first transmission delay and a second transmission delay.

[0194] Example 4.1. A user device comprising:

[0195] at least one processor; and

[0196] at least one memory storing instructions that, when executed by at least one processor, cause the user equipment to at least:

[0197] receiving a first signal from the first device, the first signal comprising first data for presentation by a second device in communication with the UE and second data for presentation by a third device in communication with the UE, the first data and the second data being synchronized in time with each other;

[0198] extracting first data from the first signal;

[0199] extracting second data from the first signal;

[0200] determining timing for presenting the first data by the second device and presenting the second data by the third device, the timing being configured to approximately synchronize presentation of the first data by the second device and presentation of the second data by the third device in time;

[0201] Sending, by the UE, a second signal including the first data to the second device; and

[0202] The UE sends a third signal including the second data to the third device,

[0203] At least one of the second signal and the third signal includes a timing-based encoded presentation delay.

[0204] Example 4.2. The user equipment of Example 4.1, wherein the second signal and the third signal are sent sequentially.

[0205] Example 4.3. The user equipment according to example 4.1 or 4.2, further comprising:

[0206] determining a first transmission delay from the UE to the second device; and

[0207] determining a second transmission delay from the UE to the third device,

[0208] Wherein the timing is based on a first transmission delay and a second transmission delay.

[0209] Example 5.1. A processor-readable medium storing instructions that, when executed by at least one processor or user equipment (UE), cause the UE to at least:

[0210] receiving, by a user equipment (UE), a first signal from a first device, the first signal comprising first data for presentation by a second device in communication with the UE and second data for presentation by a third device in communication with the UE, the first data and the second data being synchronized in time with each other;

[0211] Extracting, by the UE, first data from the first signal;

[0212] Extracting, by the UE, second data from the first signal;

[0213] Determining, by the UE, timing for presentation of the first data by the second device and presentation of the second data by the third device, the timing being configured to approximately synchronize presentation of the first data by the second device and presentation of the second data by the third device in time;

[0214] Sending, by the UE, a second signal including the first data to the second device; and

[0215] The UE sends a third signal including the second data to the third device,

[0216] Wherein at least one of the second signal or the third signal comprises a timing-based encoded presentation delay.

[0217] Example 5.2. The processor-readable medium of Example 5.1, wherein the second signal and the third signal are sent sequentially.

[0218] Example 5.3. A processor-readable medium according to any of Examples 5.1 to 5.2, wherein the instructions, when executed by at least one processor, further cause the UE to perform at least:

[0219] determining a first transmission delay from the UE to the second device; and

[0220] determining a second transmission delay from the UE to the third device,

[0221] Wherein the timing is based on a first transmission delay and a second transmission delay.

[0222] Example 6.1. A user equipment (UE), comprising:

[0223] means for receiving, by a user equipment (UE), a first signal from a first device, the first signal comprising first data for presentation by a second device in communication with the UE and second data for presentation by a third device in communication with the UE, the first data and the second data being synchronized in time with each other;

[0224] means for extracting, by the UE, first data from the first signal;

[0225] means for extracting, by the UE, second data from the first signal;

[0226] means for determining, by the UE, timing for presentation of the first data by the second device and presentation of the second data by the third device, the timing being configured to approximately synchronize presentation of the first data by the second device and presentation of the second data by the third device in time;

[0227] means for sending, by the UE, a second signal comprising the first data to a second device; and

[0228] means for sending, by the UE, a third signal comprising second data to a third device,

[0229] Wherein at least one of the second signal or the third signal comprises a timing-based encoded presentation delay.

[0230] Example 6.2. The user equipment of Example 6.1, wherein the second signal and the third signal are sent sequentially.

[0231] Example 6.3. A user equipment according to any one of Examples 6.1 to 6.2, further comprising:

[0232] means for determining a first transmission delay from the UE to the second device; and

[0233] means for determining a second transmission delay from the UE to the third device,

[0234] Wherein the timing is based on a first transmission delay and a second transmission delay.

[0235] Example 7.1. A user equipment (UE), comprising:

[0236] means for receiving a first signal from a first device, the first signal comprising first data for presentation by a second device in communication with the UE and second data for presentation by a third device in communication with the UE, the first data and the second data being synchronized in time with each other;

[0237] means for extracting first data from the first signal;

[0238] means for extracting second data from the first signal;

[0239] means for determining timing for sending a second signal comprising the first data to a second device and for sending a third signal comprising the second data to a third device, the timing being configured to approximately synchronize presentation of the first data by the second device and presentation of the second data by the third device in time;

[0240] means for sending a second signal to a second device based on the timing; and

[0241] Means for sending a third signal to a third device based on the timing.

[0242] Example 7.2. The user equipment of Example 7.1, wherein the means for determining the timing comprises means for determining processing and transmission delays of the second device.

[0243] Example 7.3. The user equipment of Example 7.1 or 7.2, wherein the means for determining the timing comprises means for determining processing and transmission delays of the third device.

[0244] Example 7.4. The user equipment of any of Examples 7.1 to 7.3, wherein sending the second signal to the second apparatus is at a first time, and sending the third signal to the third apparatus is at a second time.

[0245] Example 7.5. The user equipment of any of Examples 7.1 to 7.4, wherein sending the second signal to the second apparatus precedes sending the third signal to the third apparatus.

[0246] Example 7.6. The user device according to any one of Examples 7.1 to 7.3 also includes: a component for performing encoding of a first presentation time delay in the second signal, the first presentation time delay being configured to make the presentation of the first data approximately synchronized in time with the presentation of the second data in the third signal.

[0247] Example 7.7. The user equipment of any of Examples 7.1 to 7.6, wherein the first data comprises a first data stream and the second data comprises a second data stream.

[0248] Example 7.8. A user device according to Examples 7.1 to 7.7, wherein the first data stream is an audio data stream and the second data is a video data stream.

[0249] Example 7.9. The user equipment of any one of Examples 7.1 to 7.8, further comprising: means for decoding and processing the first data.

[0250] Example 7.10. The user equipment of any of Examples 7.1 to 7.9, further comprising: means for decoding and processing the second data.

[0251] Example 8.1. A user equipment (UE), comprising:

[0252] means for receiving a first signal including first data from a first device in communication with a UE;

[0253] means for receiving a second signal including second data from a second device in communication with the UE;

[0254] means for determining timing for synchronizing first data and second data in time;

[0255] means for generating a third signal including first data and second data based on the timing, the first data and the second data being synchronized in time in the third signal; and

[0256] Means for sending a third signal to a third device.

[0257] Example 8.2. The user equipment of Example 8.1, wherein the means for determining the timing comprises means for determining processing and transmission delays of the first device.

[0258] Example 8.3. The user equipment of Example 8.1 or 8.2, wherein the means for determining the timing comprises means for determining processing and transmission delays of the second device.

[0259] Example 8.4. The user equipment of Example 8.3, wherein the means for generating the third signal comprises means for mixing the first data of the first signal with the second data of the second signal into the third signal.

[0260] Example 8.5. A user device according to any of Examples 8.1 to 8.4, wherein the means for generating the third signal comprises means for synchronizing a presentation time of the first data with a presentation time of the second data in the third signal for presentation by a third device.

[0261] Example 8.6. The user equipment of Example 8.5, wherein the means for synchronizing presentation time of the first data with presentation time of the second data in a third signal for presentation by a third device comprises means for synchronizing data packets of the first data with data packets of the second data in time.

[0262] Example 8.7. The user equipment of any of Examples 8.1 to 8.6, wherein the first data comprises a first data stream and the second data comprises a second data stream.

[0263] Example 8.8. The user device of Example 8.7, wherein the first data stream is an audio data stream and the second data is a video data stream.

[0264] Example 8.9. A user device according to any of Examples 8.1 to 8.8, further comprising means for encoding and processing the first data.

[0265] Example 8.10. The user equipment of any of Examples 8.1 to 8.9, further comprising means for encoding and processing the second data.

[0266] The embodiments and aspects disclosed herein are examples of the present disclosure and may be embodied in various forms. For example, although some embodiments herein are described as separate embodiments, each embodiment herein may be combined with one or more other embodiments herein. The specific structural and functional details disclosed herein should not be interpreted as limitations, but as the basis for the claims, and as a representative basis for teaching those skilled in the art to use the present disclosure in various ways in almost any appropriately detailed structure. Throughout the description of the drawings, the same reference numerals may refer to similar or identical elements.

[0267] According to the present disclosure, the phrases "in one aspect", "in various aspects", "in various aspects", "in some aspects" or "in other aspects" may each refer to one or more of the same or different aspects. The phrase "plurality" may refer to two or more.

[0268] According to the present disclosure, the phrases "in one embodiment," "in an embodiment," "in various embodiments," "in some embodiments," or "in other embodiments" may each refer to one or more of the same or different embodiments. A phrase in the form "A or B" means "(A), (B), or (A and B)." A phrase in the form "at least one of A, B, or C" means "(A); (B); (C); (A and B); (A and C); (B and C); or (A, B, and C)."

[0269] Any of the methods, programs, algorithms or codes described herein can be converted into a programming language or a computer program, or represented by it. The terms "programming language" and "computer program" used herein each include any language for specifying computer instructions, and include (but are not limited to) the following languages ​​and their derivatives: assembly language, Basic, batch files, BCPL, C, C++, Delphi, Fortran, Java, JavaScript, machine code, operating system command language, Pascal, Perl, PL1, Python, scripting languages, Visual Basic, metalanguages ​​that specify programs themselves, and all first, second, third, fourth, fifth or subsequent computer languages. Databases and other data models, and any other metalanguages ​​are also included. There is no difference between the languages ​​of interpreting, compiling or using both compiling and interpreting methods. There is no difference between the compiled version and the source version of the program. Therefore, in a programming language, it can exist in more than one state (such as source, compiled, object or link), and a reference to a program is a reference to any and all such states. A reference to a program can cover the actual instructions and / or the intentions of these instructions.

[0270] Although various aspects of the present disclosure are shown in the accompanying drawings, it is not intended that the present disclosure is limited thereto, as the scope of the present disclosure should be as broad as possible, and the specification should be read in the same manner. Therefore, the above description should not be interpreted as limiting, but only as an example of specific aspects. Those skilled in the art will envision other modifications within the scope and spirit of the appended claims.

Claims

1. A communication method, the method comprising: A first signal is received by a wireless communication function of a user equipment (UE) from a network device of a network system, wherein the first signal includes: first data for presentation by a peripheral device associated with the UE and in communication with the UE, and second data for presentation by another peripheral device associated with the UE and in communication with the UE, the first data and the second data being synchronized with each other in time; extracting, by a media access function (MAF) of the UE, the first data from the first signal; extracting, by the MAF of the UE, the second data from the first signal; determining, by the MAF of the UE, timing for sending a second signal including the first data to the peripheral device and for sending a third signal including the second data to the other peripheral device, the timing being configured to approximately synchronize presentation of the first data by the peripheral device and presentation of the second data by the other peripheral device in time; transmitting, by the wireless communication function of the UE, the second signal to a peripheral device based on the timing; and The third signal is transmitted to the other peripheral device by the wireless communication function of the UE based on the timing.

2. The method according to claim 1, further comprising: determining processing and transmission delays of at least said peripheral device, Wherein said determining said timing comprises: determining said timing based at least on said processing and transmission delays of said peripheral device.

3. The method according to any one of claims 1 to 2, further comprising: determining processing and transmission delays of the another peripheral device, Wherein said determining said timing comprises: determining said timing based at least on said processing and transmission delays of said another peripheral device.

4. The method according to any one of claims 1 to 3, wherein: Sending the second signal to the peripheral device based on the timing includes: sending the second signal to the peripheral device at a first time, and Transmitting the third signal to the other peripheral device based on the timing includes transmitting the third signal to the other peripheral device at a second time different from the first time. The method of claim 4 , wherein the first time precedes the second time.

6. The method according to any one of claims 1 to 3, further comprising: A first presentation time offset is encoded in the second signal based on the timing.

7. The method according to any one of claims 1 to 6, wherein the first data comprises a first data stream and the second data comprises a second data stream.

8. The method of claim 7, wherein the first data stream is one of an audio data stream or a video data stream, and the second data is the other of the audio data stream and the video data stream.

9. The method according to any one of claims 1 to 8, further comprising: The first data is decoded and processed.

10. The method according to any one of claims 1 to 9, further comprising: The second data is decoded and processed.

11. A communication method, the method comprising: receiving, by a wireless communication function of a user equipment (UE), a first signal including first data from a first peripheral device of the UE in communication with the UE; receiving, by the wireless communication function of the UE, a second signal including second data from a second peripheral device of the UE in communication with the UE; Determining timing by a media access function (MAF) of the UE, the timing synchronizing the first data and the second data in time; generating, by the MAF of the UE, a third signal including the first data and the second data based on the timing, wherein the first data and the second data are synchronized in time in the third signal; as well as The third signal is sent by the wireless communication function to a network device of the network system.

12. The method of claim 11, wherein said determining said timing comprises: Processing and transmission delays of the first peripheral device are determined.

13. The method according to any one of claims 11 to 12, wherein said determining said timing comprises: Processing and transmission delays of the second peripheral device are determined.

14. The method of claim 13, wherein said generating said third signal comprises: The first data of the first signal and the second data of the second signal are mixed into the third signal.

15. The method according to any one of claims 11 to 14, wherein said generating said third signal comprises: A presentation time of the first data is synchronized with a presentation time of the second data in the third signal.

16. The method of claim 15, wherein said synchronizing the presentation time of the first data with the presentation time of the second data in the third signal comprises: The data packets of the first data are synchronized in time with the data packets of the second data.

17. The method according to any one of claims 11 to 16, wherein the first data comprises a first data stream and the second data comprises a second data stream.

18. The method of claim 17, wherein the first data stream is an audio data stream and the second data is a video data stream.

19. The method according to any one of claims 11 to 18, further comprising: The first data is encoded and processed.

20. The method according to any one of claims 11 to 19, further comprising: The second data is encoded and processed.

21. A user equipment (UE), comprising: at least one processor; as well as At least one memory storing instructions, which, when executed by the at least one processor, cause the user equipment to at least perform the method according to any one of claims 1 to 20.

22. A processor-readable medium storing instructions, which, when executed by at least one processor of a device, cause the device to at least perform the method according to any one of claims 1 to 20.

Images