Synchronization of multimodal services

By determining the correlation between multiple data streams in the communication device and utilizing the super PDU set and time window mechanism, the problem of data stream synchronization in multimodal services is solved, realizing the synchronous transmission of multimodal services in the 5G system and improving the user experience.

CN123077136APending Publication Date: 2026-11-10ALCATEL LUCENT SHANGHAI BELL CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202480090597.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-04-03
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing technologies struggle to effectively synchronize different data streams in multimodal services, leading to a decline in user experience. This is especially true in 5G systems, where the asynchronous nature of haptic and video streams and their varying QoS requirements make it difficult to achieve synchronized transmission using existing multimodal service IDs.

Method used

By determining the association between multiple data streams in the communication device, utilizing the concept of a super PDU set, multiple data streams are processed based on association, including mapping the association between QoS streams and DRBs, and using time windows and timer mechanisms to ensure the synchronous transmission of data streams, thus achieving synchronization of multimodal services.

Benefits of technology

It improves the synchronization flexibility and effectiveness of multimodal services, ensures the synchronous transmission of different modal data streams within the synchronization threshold, and enhances the user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123077136A_ABST
    Figure CN123077136A_ABST
Patent Text Reader

Abstract

Embodiments of this disclosure relate to the synchronization of multimodal services. In one aspect, a communication device determines an association between multiple data streams. Furthermore, the communication device processes the multiple data streams based on this association.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Various example embodiments relate to the field of communications, and more particularly to devices, methods, apparatuses, and computer-readable storage media for synchronizing multimodal services. Background Technology

[0002] A communication network can be viewed as a facility that enables communication between two or more communication devices or provides communication devices with access to a data network. Mobile or wireless communication networks are an example of communication networks.

[0003] Such communication networks operate according to standards such as those issued by 3GPP (3rd Generation Partnership Project) or ETSI (European Telecommunications Standards Institute). Examples of such standards include the so-called 5G (fifth generation) standard or other standards issued by 3GPP. Summary of the Invention

[0004] Generally, the exemplary embodiments of this disclosure provide solutions related to the synchronization of multimodal services. A multimodal service is a communication service consisting of several data streams that are related to each other and coordinated by an application. The data streams can transmit different types of data (e.g., audio, video, location, haptic data).

[0005] In a first aspect, a communication device is provided. The communication device includes at least one processor and at least one memory, the at least one memory storing instructions that, when executed by the at least one processor, cause the communication device to at least: determine an association between a plurality of data streams; and, based on the association, process the plurality of data streams.

[0006] In a second aspect, a method is provided. The method includes determining an association between multiple data streams at a communication device; and processing the multiple data streams based on the association at the communication device.

[0007] In a third aspect, an apparatus is provided. The apparatus includes components for determining an association between a plurality of data streams at a communication device; and components for processing the plurality of data streams based on the association at the communication device.

[0008] In a fourth aspect, a non-transitory computer-readable medium is provided, the non-transitory computer-readable medium including program instructions for causing the apparatus to at least perform the method according to the second aspect above.

[0009] In a fifth aspect, a computer program comprising instructions is provided, which, when executed by a device, cause the device to perform at least the method according to the second aspect above.

[0010] In a sixth aspect, a communication device is provided. The communication device includes a determining circuit system configured to determine an association between a plurality of data streams, and a processing circuit system configured to process the plurality of data streams based on the association.

[0011] It should be understood that the summary portion is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] Some exemplary embodiments will now be described with reference to the accompanying drawings, in which: Figure 1A An example environment in which example embodiments of this disclosure may be implemented is shown; Figure 1B Examples of multimodality in 5G XR services associated with some embodiments of this disclosure are shown; Figure 1C and 1D Two options for haptic and video streaming modeling associated with some embodiments of this disclosure are shown; Figure 1E Example typical synchronization thresholds for immersive multimodal virtual reality (VR) applications in a 5G system associated with some embodiments of this disclosure are shown; Figure 2 A flowchart is shown illustrating a method implemented at a communication device according to some embodiments of the present disclosure; Figure 3 An example set of Hyperprotocol Data Units (PDUs) according to some embodiments of this disclosure is shown; Figure 4 A simplified block diagram of a device suitable for implementing some example embodiments of this disclosure is shown; and Figure 5 A block diagram illustrating an example of a computer-readable medium according to some exemplary embodiments of the present disclosure is shown.

[0013] In all the accompanying drawings, the same or similar reference numerals denote the same or similar elements. Detailed Implementation

[0014] The principles of this disclosure will now be described with reference to some exemplary embodiments. It should be understood that these embodiments are described for illustrative purposes and to assist those skilled in the art in understanding and implementing this disclosure, and do not imply any limitation on the scope of this disclosure. The disclosure described herein can be implemented in various ways other than those described below.

[0015] In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0016] References to "an embodiment," "embodiment," "example embodiment," etc., in this disclosure indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment necessarily includes that particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. Additionally, when a particular feature, structure, or characteristic is described in connection with an embodiment, whether explicitly described or not, the effect of combining it with other embodiments on such feature, structure, or characteristic is considered to be within the knowledge of those skilled in the art.

[0017] It should be understood that although the terms “first” and “second”, etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, without departing from the scope of the exemplary embodiments, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. As used herein, the term “and / or” includes any and all combinations of one or more of the listed terms.

[0018] The terminology used herein is for the purpose of describing particular embodiments and is not intended to limit the exemplary embodiments. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that, when used herein, the terms “comprises,” “comprising,” “has,” “having,” “includes,” and / or “including” specify the presence of stated features, elements, and / or components, but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof. As used herein, “at least one of the following: ” and “at least one of ” and similar wording (where the list of two or more elements is connected by “and” or “or”) means at least any one of the elements, or at least any two or more of the elements, or at least all of the elements.

[0019] As used in this application, the term "circuit" may refer to one or more of the following: (a) Hardware circuit implementation only (such as implementation only in analog and / or digital circuits) and (b) A combination of hardware circuitry and software, such as (if applicable): (i) A combination of (multiple) analog and / or digital hardware circuits and software / firmware, and (ii) Any part of a hardware processor(s) having software (including (multiple) digital signal processors), software, and (multiple) memories, which work together to enable a device such as a mobile phone or server to perform various functions, and (c) (Multiple) hardware circuits and / or (multiple) processors, such as (multiple) microprocessors or a portion thereof, which require software (e.g., firmware) to operate, but may be absent when the software is not required to operate.

[0020] This definition of "circuit" applies to all uses of the term in this application (including in any claim). As another example, as used in this application, the term "circuit" also covers implementations of hardware circuitry or processors (or processors in general) and their accompanying software and / or firmware. The term "circuit" also covers, for example and if applicable to a particular claim element, baseband integrated circuits or processor integrated circuits for mobile devices or similar integrated circuits in servers, cellular network devices, or other computing or network devices.

[0021] As used herein, the term "communication network" refers to a network that conforms to any suitable communication standard, such as New Radio (NR), Long Term Evolution (LTE), LTE-A Advanced (LTE-A), Wideband Code Division Multiple Access (WCDMA), High-Speed ​​Packet Access (HSPA), Narrowband Internet of Things (NB-IoT), etc. Furthermore, communication between terminal devices and network devices in the communication network can be performed according to any suitable generation of communication protocols, including but not limited to third-generation (3G), 4G, 4.5G, 5G, or 6G communication protocols and / or any other currently known or future-developed protocols. Embodiments of this disclosure can be applied to various communication systems. Given the rapid development of communications, there will certainly be future types of communication technologies and systems that embody the nature of this disclosure. This should not be construed as limiting the scope of this disclosure to the aforementioned systems.

[0022] As used herein, the term "network device" refers to a node in a communication network through which terminal devices can access the network and receive services. Network devices can refer to base stations (BS) or access points (APs), such as Node B (NodeB or NB), Radio Access Network (RAN) nodes, Evolved Node B (eNodeB or eNB), NR NB (also known as gNB), Remote Radio Unit (RRU), Radio Header (RH), infrastructure equipment for V2X (Vehicle-to-Everything) communication, Transmit and Receive Point (TRP), Receive Point (RP), Remote Radio Header (RRH), relay, Integrated Access and Backhaul (IAB) nodes, low-power nodes such as femtoBS, picoBS, etc., depending on the terminology and technology used.

[0023] The term "terminal device" refers to any terminal device capable of wireless communication. By way of example and not limitation, a terminal device may also be referred to as a communication device, user equipment (UE), subscriber station (SS), portable subscriber station, mobile station (MS), or access terminal (AT). Terminal devices may include, but are not limited to, mobile phones, cellular phones, smartphones, Voice over IP (VoIP) phones, wireless local loop phones, tablets, wearable terminal devices, personal digital assistants (PDAs), portable computers, desktop computers, image capture terminal devices such as digital cameras, gaming terminal devices, music storage and recycle bins, in-vehicle wireless terminal devices, wireless endpoints, mobile stations, laptop embedded devices (LEEs), laptop-mounted devices (LMEs), USB dongles, smart devices, wireless customer premises equipment (CPEs), Internet of Things (IoT) devices, watches or other wearable devices, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in the context of industrial and / or automated processing chains), consumer electronics devices, devices operating on commercial and / or industrial wireless networks, etc. In the following description, the terms "terminal equipment", "communication equipment", "terminal", "user equipment" and "UE" are used interchangeably.

[0024] The principles and embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. First, refer to... Figure 1A It illustrates an example environment 100 in which example embodiments of the present disclosure may be implemented.

[0025] Environment 100, which may be part of a communication network, includes terminal devices 110 and network devices 120 communicating with each other. Communication between terminal devices 110 and network devices 120 can be direct or indirect. As an example, terminal devices 110 and network devices 120 can... Figure 1ACommunicates with one or more other devices not shown in the diagram.

[0026] To transmit data and / or control information, terminal device 110 may perform communication with network device 120. The link from network device 120 to terminal device 110 is referred to as a downlink (DL), while the link from terminal device 110 to network device 120 is referred to as an uplink (UL).

[0027] Despite Figure 1A The communication environment 100 describes terminal device 110 and network device 120, but the embodiments of this disclosure can be equivalently applied to any other suitable communication devices communicating with each other. That is, the embodiments of this disclosure are not limited to... Figure 1A An exemplary scenario. In this regard, it should be noted that, although in Figure 1A The terminal device is schematically depicted as a mobile phone and the network device 120 is schematically depicted as a base station; however, it should be understood that these depictions are exemplary in nature and do not imply any limitation. In other embodiments, the first device 110 and the network device 120 may be any other communication device, such as any other wireless communication device.

[0028] It should be understood that, such as Figure 1A The specific numbers of various communication devices and communication links shown are for illustrative purposes only and do not imply any limitation. Communication environment 100 may include any suitable number of communication devices and any suitable number of communication links for implementing embodiments of this disclosure. Furthermore, it should be understood that various wireless and wired communications may exist between all communication devices (if desired).

[0029] Communications in Environment 100 may follow any suitable communication standards or protocols that are already in use or will be developed in the future, such as Universal Mobile Telecommunications System (UMTS), Long Term Evolution (LTE), LTE-A Advanced, 5G New Radio (NR), 6G, Wi-Fi, and Global Microwave Access Interoperability (WiMAX) standards, and employ any suitable communication technologies, including, for example, Multiple-Input Multiple-Output (MIMO), Orthogonal Frequency Division Multiplexing (OFDM), Time Division Multiplexing (TDM), Frequency Division Multiplexing (FDM), Code Division Multiplexing (CDM), Bluetooth, ZigBee and Machine-Type Communications (MTC), Enhanced Mobile Broadband (eMBB), Massive Machine-Type Communications (mMTC), Ultra-Reliable Low-Latency Communications (URLLC), Carrier Aggregation (CA), Dual Connectivity (DC), and New Radio Unlicensed (NR-U) technologies.

[0030] In the 5G era, mobile media services, cloud-based augmented reality (AR) / virtual reality (VR), cloud gaming (CG), and video-based remote control for machines or drones are expected to contribute increasing traffic to 5G networks. Extended reality (XR) and media services supporting 5G will be further enhanced to support the metaverse. The research on XR and media services in Release 18 (Rel-18) originates from "Haptic and Multimodal Communication Services" as defined in 3GPP Technical Report (TR) 22.847.

[0031] In multimodal interaction systems, a modality is a type of information or representation within a specific interaction system. Modal representations include video, audio, and haptic feedback (vibrations or other movements that provide tactile sensation). For typical multimodal communication services / applications, different modalities exist that influence the user experience, for example: - Video / audio media; - Information about the environment sensed by sensors, such as brightness, temperature, humidity, etc.; - Tactile data, such as the sensation of touching a surface (e.g., pressure, texture, vibration, temperature), or kinesthetic data (e.g., gravity, tension, position perception).

[0032] Figure 1B Examples of multimodality in 5G XR services associated with some embodiments of this disclosure are shown. For example... Figure 1B As shown, multiple modalities (e.g., video media from glasses and tactile data from gloves) can be transmitted simultaneously to one or more application servers for further processing in a coordinated manner in terms of Quality of Service (QoS) coordination, service synchronization, energy saving, etc.

[0033] Furthermore, a new service, haptic streaming, has been identified in 3GPP TR 22.856. Haptic streaming is time-sensitive, and its arrival is aperiodic. The arrival of haptic packets and video packets is asynchronous, and the density of haptic packets is much higher than that of video packets. Figure 1C and Figure 1D Two options for haptic and video stream modeling associated with some embodiments of this disclosure are shown, wherein Figure 1C Option 1, showing the arrival of tactile groups in the time domain following a Pareto distribution, is illustrated. Figure 1D Option 2 is shown, which involves tactile grouping (e.g., with a large number of sensors) in each time slot.

[0034] Additionally, 3GPP Technical Specification (TS) 23.501, as a protocol, has introduced a Multimodal Service Identifier (MMSID) for multiple IP data streams associated with a multimodal service. The MMSID is used to define the association between multiple IP data streams within a multimodal service.

[0035] Because multiple media components are processed separately, synchronization between these components is crucial to avoid negatively impacting user experience. The aforementioned multimodal service ID only relates to the association between multiple IP data streams, but does not define the association between each packet of different data streams. Therefore, it is difficult to achieve synchronized transmission of multimodal service packets based solely on the multimodal service ID.

[0036] Furthermore, the multimodal synchronization threshold can be defined as the maximum tolerable time interval between the initiation of two stimuli, where one stimulus is presented to one sense and another to another sense, such that the accompanying sensory objects are perceived as synchronized. As stated in 3GPP TS 22.261, applying a synchronization threshold in 5G systems can help support immersive multimodal applications when the synchronization threshold between two or more modalities is less than the application's latency key performance indicator (KPI). Figure 1E Example typical synchronization thresholds for immersive multimodal VR applications in a 5G system as described in 3GPP TS 22.261, associated with some embodiments of this disclosure, are shown. However, the synchronization thresholds are only considered as the worst-case requirements for per-packet synchronization within a multimodal service. They do not indicate how the actual operations for synchronization should be performed.

[0037] Given the above, optimizing the synchronization of multimodal services remains a significant problem to be solved. Specifically, feasible mechanisms need to be designed within the RAN (e.g., between the UE and gNB) to ensure the synchronized transmission of associated packets within multimodal services. Furthermore, the inventors have noted that assuming QoS flows for the same multimodal service ID are mapped to the same data radio bearer (DRB) is unreasonable, considering the different QoS requirements of different modalities. Therefore, some components for synchronization between different DRBs also need to be considered.

[0038] According to embodiments of this disclosure, a scheme for synchronizing multimodal services is provided. Using this scheme, a communication device determines the association between multiple data streams. Furthermore, the communication device processes the multiple data streams based on the association.

[0039] By processing multiple data streams in association, synchronization of multiple data streams can be facilitated. In this way, the flexibility and efficiency of synchronization for multimodal services can be improved.

[0040] Figure 2 A flowchart 200 illustrates a method implemented at a communication device according to some embodiments of the present disclosure. For purposes of discussion, reference will be made to... Figure 1A Method 200 is described. Considering both UL and DL services, the communication equipment may include terminal equipment 110 or network equipment 120.

[0041] like Figure 2 As shown, at box 210, the communication device determines the association between multiple data streams. For example, the multiple data streams may be associated with multiple types of services within a multimodal service. The multiple data streams may be associated with a multimodal service ID of the multimodal service. The multiple data streams may include one or more of video data streams, audio data streams, or haptic data streams. It should be understood that any other suitable service type may be involved, and the scope of this disclosure is not limited in this respect.

[0042] In some example embodiments, associations between multiple data flows may include associations between multiple Quality of Service (QoS) flows mapped from the multiple data flows. For example, multiple QoS flows may be configured with the same multimodal service ID to indicate association. The multimodal service ID may be configured / visible or transparent to end device 110. Alternatively or additionally, associations between multiple data flows may include associations between multiple data radio bearers (DRBs) mapped from the multiple QoS flows of the multiple data flows. For example, based on the mapping between QoS flows and DRBs and the multimodal service IDs of the QoS flows, network device 120 may configure end device 110 which DRBs need to be considered for the aforementioned associations.

[0043] In some example implementations, determining the association between multiple data streams may include identifying one or more associated PDUs across the multiple data streams. In some implementations, the concept or term "super PDU set" may be introduced for multimodal synchronization to define a set of PDUs that includes one or more of the aforementioned associated PDUs across multiple data streams. In other words, a super PDU set may include one or more associated PDUs across multiple QoS streams or DRBs. A super PDU set may also be referred to as a multimodal synchronization PDU set (MMS-PS). It should be understood that a super PDU set may also be referred to by any other name, and the scope of this disclosure is not limited in this respect. Each super PDU set may be identified by a unique sequence number (SN). The ordering of multiple SNs across multiple super PDU sets may be based on the ordering of PDUs from one of the multiple data streams.

[0044] In some example embodiments, the communication device can determine the association between multiple data streams (i.e., a super PDU set) based on one or more of the following: the DRB association configuration described above (if any), the multimodal service ID (if visible), the burst arrival time, and jitter(s) of one of the multiple data streams. As an example, at the data stream level, multiple data streams can be identified as associated based on the DRB association configuration described above (if any) or the multimodal service ID (if visible). At the PDU level, the super PDU set can be determined in various ways.

[0045] In some example embodiments where multiple data streams include at least a first data stream and a second data stream, the communication device may determine a time window based on the arrival time and margin (also referred to as margin range, time range, or range) of the PDUs in the first data stream. In this case, the first data stream may be used as a baseline. As an example, a video data stream may be used as a baseline. For example, the margin may be determined based on the jitter(s) of the video data stream(s). Then, based on the time window, the communication device may determine a super PDU set, which includes one or more PDUs from the first data stream whose arrival time is within the time window, and one or more PDUs from the second data stream whose arrival time is within the time window.

[0046] refer to Figure 3 Let's discuss the example super PDU set and its determination process. Figure 3 As shown, video frames arrive periodically every 1 / F seconds, where F represents the frame rate in frames per second (FPS), for example, F = 30, 60, or 90 FPS. The communication device can select a video frame (or a PDU of a video frame) as an anchor point. Then, centered on this anchor point, a time window can be defined as the arrival time of the anchor point ± x ms. In this case, the margin is set to ±4 ms. Alternatively or additionally, the margin can be set to any other value. Based on this, there is a correlation between the arrival times of the PDUs within the time window (i.e., the anchor video frame / PDU and the haptic packet PDU). These PDUs (i.e., the anchor video frame / PDU and the haptic packet PDU) form a super PDU set. In other words, as... Figure 3 As shown, at the moment when the anchor video frame F arrives at the buffer of the communication device (i.e., the terminal device 110 for the uplink or the network device 120 for the downlink), the haptic frames preceding and following the anchor video frame can form a super PDU set together. Alternatively or additionally, if other modalities exist, one or more PDUs of other modalities arriving within this time window can also be included in the super PDU set, for example, audio data.

[0047] In some example embodiments where multiple data streams include at least a first data stream and a second data stream, the communication device can determine a time window with a width. The time window can be continuous and non-overlapping. In other words, a sliding window with a specific width can be determined, independent of the anchor point in the previous example. Based on this, a super PDU set can be determined to include at least one of the following: one or more PDUs from the first data stream whose arrival time is within the time window, or one or more PDUs from the second data stream whose arrival time is within the time window.

[0048] It should be understood that although only the first data stream and the second data stream are discussed in some example embodiments, the multiple data streams may include any number of data streams, and the scope of this disclosure is not limited in this respect.

[0049] Return to reference Figure 2 After determining the associations between multiple data streams, at box 220, the communication device processes the multiple data streams based on the associations. In some example embodiments, the communication device may process the multiple data streams based on both associations and time requirements. For example, the time requirements may be determined based on a synchronization threshold for a multimodal service.

[0050] In some example embodiments, when the time requirement is met, the communication device can transmit one or more PDUs of the super PDU set to another communication device (i.e., the peer node). As an example, it would be ideal if the super PDU set could be transmitted to the peer node when the time requirement is met, enabling synchronization of the PDUs in the super PDU set.

[0051] In an example embodiment of the communication device including terminal device 110 for UL transmission, terminal device 110 may transmit a request to network device 120 for uplink authorization for transmitting a super PDU set. For example, the request may include the SN of the super PDU set. Therefore, network device 120 may identify the super PDU set and schedule uplink authorization taking into account the synchronization of the super PDU set. Network device 120 may then transmit the uplink authorization to terminal device 110.

[0052] In some example implementations, the communication device can transmit time requirements to peer nodes. In this case, multiple data streams can be processed by the peer node based on the received time requirements. For example, for UL transmissions, the aforementioned uplink authorization can be determined by network device 120 based on the received time requirements.

[0053] In some example implementations, for UL transmission, timing requirements can be configured for each DRB with multimodal synchronization requirements to the terminal device 110 to ensure that (multiple) PDUs in the super PDU set can be transmitted within the latency requirements.

[0054] In some example embodiments, the communication device can handle PDU dropping for multiple data streams based on time requirements. For example, after a time requirement is not met, the communication device can drop one or more remaining PDUs (if any) from a super PDU set. In some example implementations, similar to the PDU Set Integration Processing Instruction (PSIHI), per-PDU set dropping can also be applied to per-super PDU set dropping, if configured.

[0055] In some example embodiments, the communication device can process a logical channel prioritization process for multiple data streams based on time requirements. For example, a super PDU set can take precedence over one or more other PDUs because they need to be processed according to time requirements.

[0056] In some example embodiments, the time requirement can be associated with a timer running at the communication device as a time limit for processing multiple data streams. In this case, the timer setting can be based on a synchronization threshold for the multimodal service. For example, a timer can be maintained for each PDU in a super PDU set, and the timer can be started when any data in the super PDU set (i.e., any PDU) is successfully transmitted. When the timer for the super PDU set expires, any remaining data in the super PDU set can be discarded. For example, for UL transmissions, terminal device 110 can assume successful PDU transmission after adding a margin (e.g., one retransmission time) to the time used to transmit a PDU. As another example, for DL ​​transmissions, network device 120 can determine successful PDU transmission after receiving an acknowledgment (ACK) for the PDU from terminal device 110.

[0057] As an example implementation, the timing requirements can be determined based on the most stringent timing requirements of multiple data streams in the multimodal service. For example, a timer can be set based on the most stringent typical multimodal synchronization threshold of multiple data streams. In other words, a timer can be set using a time length within which any PDU in the super PDU set is transmitted according to the minimum latency requirements of the multiple data streams in the multimodal service. Based on the identifier of the super PDU set, the timer setting for the super PDU set, and the frames of each data stream transmitted sequentially, it is possible to ensure sequential and in-window synchronization of the multimodal service; that is, all PDUs of different modalities of the multimodal service can be transmitted based on the timing requirements.

[0058] To differentiate the different processing of different data streams relative to each other's varying time requirements (e.g., synchronization thresholds), a more precise determination of the time requirements can be given. In an example implementation where multiple data streams include at least a third and a fourth data stream, if the PDU in the super PDU set that is first successfully transmitted to the peer node is a PDU of the third data stream, the time requirement can be determined based on the latency requirement of the fourth data stream compared to the third data stream.

[0059] As an example implementation, consider the following: Figure 1EThe typical synchronization thresholds for immersive multimodal VR applications in the 5G system shown differ for each media component, with the latency requirement varying relative to other media components. For example, the synchronization threshold for visual services being delayed relative to haptic services is 15ms, but the synchronization threshold for haptic services being delayed relative to visual services could be as high as 50ms. Timers can be dynamically adjusted based on the successful delivery status of PDUs in the super PDU set. Accordingly, scheduling for the remaining PDUs will be based on the set timers.

[0060] After determining such Figure 3 In the example implementation of the example superPDU set shown, if video frame F is scheduled first and successfully transmitted at time T, the timer for the haptic PDU in the superPDU set can be updated based on a synchronization threshold for the delay of the haptic packet relative to the video frame, i.e., Timer = T + ST1. Otherwise, if the haptic PDU is scheduled first and successfully transmitted at time T, the timers for video frame F and other haptic PDUs in the superPDU set can be updated based on a synchronization threshold for the delay of the video frame relative to the haptic packet, i.e., Timer = T + ST2. ST1 and ST2 can refer to different cases where one component is delayed relative to another. For example, ST1 = 50 ms and ST2 = 15 ms.

[0061] The timer can have a variety of names. For example, the timer can be simply called a super PDU set discard timer. We can also refer to the concept as the super PDU set delay budget, the super PDU set remaining time, etc.

[0062] In some example embodiments, the aforementioned PDU may include at least one of the following: a Service Data Unit (SDU) of the Packet Data Convergence Protocol (PDCP) layer, a PDU of the PDCP layer, an SDU of the Service Data Access Protocol (SDAP) layer, or a PDU of the SDAP layer. In other words, PDU dropping can occur at the PDCP layer or the SDAP layer.

[0063] According to the reference Figure 2 and Figure 3 Some embodiments, based on the synchronous processing of different media types multiplexed within a multimodal XR data stream, are allowed to ensure that associated PDUs of different modalities are transmitted to peer nodes within a synchronization range limited by a synchronization threshold.

[0064] In some example embodiments, a method implemented at a communication device is provided. The method includes determining an association between multiple data streams; and processing the multiple data streams based on the association.

[0065] In some example implementations, multiple data streams are associated with multiple types of services within a multimodal service.

[0066] In some example implementations, multiple data streams are associated with a multimodal service identifier for a multimodal service.

[0067] In some example embodiments, the association between multiple data streams includes one of the following: an association between multiple Quality of Service (QoS) streams mapped from multiple data streams; or an association between multiple Data Radio Bearers (DRBs) mapped from multiple QoS streams of multiple data streams.

[0068] In some example embodiments, determining the association between multiple data streams includes determining the association between multiple data streams based on the arrival time of one or more Protocol Data Units (PDUs) of the multiple data streams.

[0069] In some example embodiments, the multiple data streams include at least a first data stream and a second data stream, and determining the association between the multiple data streams includes: determining a time window based on margin and the arrival time of PDUs in the first data stream; determining a super PDU set based on the time window, the super PDU set including one or more PDUs in the first data stream whose arrival time is within the time window, and one or more PDUs in the second data stream whose arrival time is within the time window.

[0070] In some example embodiments, the multiple data streams include at least a first data stream and a second data stream, and determining the association between the multiple data streams includes: determining a time window with a width, wherein the time windows are continuous and non-overlapping; determining a super PDU set based on the time window, the super PDU set including at least one of the following: one or more PDUs of the first data stream whose arrival time is within the time window, or one or more PDUs of the second data stream whose arrival time is within the time window.

[0071] In some example embodiments, determining the association between multiple data streams includes determining a super PDU set, which includes one or more associated PDUs across the multiple data streams.

[0072] In some example embodiments, processing multiple data streams includes at least one of the following: processing a logical channel prioritization process for the multiple data streams; or processing PDU dropping for the multiple data streams. In some example embodiments, processing multiple data streams based on association is also based on time requirements. In some example embodiments, the time requirements are determined according to a synchronization threshold for a multimodal service. In some example embodiments, the multiple data streams include at least a third data stream and a fourth data stream, the PDU of one or more associated PDUs of the multiple data streams that is first successfully transmitted to another communication device is the PDU of the third data stream, and the time requirements are determined based on a latency requirement for the fourth data stream compared to the third data stream. In some example embodiments, the time requirements are transmitted to the other communication device. In some example embodiments, the multiple data streams are processed by the other communication device based on the received time requirements. In some example embodiments, the communication device is a terminal device, and the other communication device is a network device, and the method further includes: transmitting to the network device a request for uplink authorization, the uplink authorization being used to transmit a super PDU set, the super PDU set including one or more associated PDUs of multiple data streams, the request including a sequence number (SN) of the super PDU set; and receiving uplink authorization from the network device, the uplink authorization being determined based on time requirements.

[0073] In some example embodiments, processing multiple data streams based on association further includes: transmitting one or more PDUs from one or more associated PDUs of the multiple data streams to another communication device when a time requirement is met. In some example embodiments, processing multiple data streams based on association further includes: discarding one or more remaining PDUs from one or more associated PDUs of the multiple data streams after a time requirement has not been met. In some example embodiments, one or more associated PDUs are one or more PDUs from a super PDU set determined when determining the association between the multiple data streams.

[0074] In some example implementations, the time requirement is associated with a timer.

[0075] In some example embodiments, the time requirement is associated with a timer, and the method further includes starting the timer based on determining that any of the one or more associated PDUs has been successfully transmitted.

[0076] In some example embodiments, a PDU includes at least one of the following: a Service Data Unit (SDU) of the Packet Data Convergence Protocol (PDCP) layer; a PDU of the PDCP layer; an SDU of the Service Data Access Protocol (SDAP) layer; or a PDU of the SDAP layer.

[0077] In some example embodiments, the multiple data streams include at least a video data stream.

[0078] In some example embodiments, the apparatus capable of performing method 200 (e.g., communication) may include components for performing corresponding steps of method 200. These components may be implemented in any suitable form. For example, the component may be implemented as a circuit or a software module.

[0079] In some example embodiments, the apparatus includes components for determining associations between multiple data streams; and components for processing the multiple data streams based on the associations.

[0080] In some example implementations, multiple data streams are associated with multiple types of services within a multimodal service.

[0081] In some example implementations, multiple data streams are associated with a multimodal service identifier for a multimodal service.

[0082] In some example embodiments, the association between multiple data streams includes one of the following: an association between multiple Quality of Service (QoS) streams mapped from multiple data streams; or an association between multiple Data Radio Bearers (DRBs) mapped from multiple QoS streams of multiple data streams.

[0083] In some example embodiments, the component for determining the association between multiple data streams includes: a component for determining the association between multiple data streams based on the arrival time of one or more Protocol Data Units (PDUs) of the multiple data streams.

[0084] In some example embodiments, the plurality of data streams include at least a first data stream and a second data stream, and the components for determining the association between the plurality of data streams include: components for determining a time window based on margin and the arrival time of PDUs of the first data stream; and components for determining a super PDU set based on the time window, the super PDU set including one or more PDUs of the first data stream whose arrival time is within the time window, and one or more PDUs of the second data stream whose arrival time is within the time window.

[0085] In some example embodiments, the plurality of data streams includes at least a first data stream and a second data stream, and the components for determining the association between the plurality of data streams include: components for determining a time window having a width, wherein the time window is continuous and non-overlapping; and components for determining a super PDU set based on the time window, the super PDU set including at least one of the following: one or more PDUs of the first data stream whose arrival time is within the time window, or one or more PDUs of the second data stream whose arrival time is within the time window.

[0086] In some example embodiments, the components for determining the association between multiple data streams include: components for determining a super PDU set, which includes one or more associated PDUs across the multiple data streams.

[0087] In some example embodiments, the components for processing multiple data streams include at least one of the following: components for processing a logical channel prioritization process for the multiple data streams; or components for processing PDU discarding for the multiple data streams. In some example embodiments, processing multiple data streams based on association is also based on time requirements. In some example embodiments, the time requirements are determined according to a synchronization threshold for a multimodal service. In some example embodiments, the multiple data streams include at least a third data stream and a fourth data stream, the PDU of one or more associated PDUs of the multiple data streams that is first successfully transmitted to another communication device is the PDU of the third data stream, and the time requirements are determined based on a latency requirement for the fourth data stream compared to the third data stream. In some example embodiments, the time requirements are transmitted to the other communication device. In some example embodiments, the multiple data streams are processed by the other communication device based on the received time requirements. In some example embodiments, the communication device is a terminal device, and the other communication device is a network device, and the apparatus further includes components for transmitting to the network device a request for uplink grant, the uplink grant being used to transmit a super PDU set, the super PDU set comprising one or more associated PDUs of multiple data streams, the request including a sequence number (SN) of the super PDU set; and components for receiving uplink grant from the network device, the uplink grant being determined based on time requirements.

[0088] In some example embodiments, the components for processing multiple data streams based on association further include: components for transmitting one or more PDUs from one or more associated PDUs of the multiple data streams to another communication device when a time requirement is met. In some example embodiments, the components for processing multiple data streams based on association further include: components for discarding one or more remaining PDUs from one or more associated PDUs of the multiple data streams after a time requirement has not been met. In some example embodiments, the one or more associated PDUs are one or more PDUs from a super PDU set determined when determining the association between the multiple data streams.

[0089] In some example implementations, the time requirement is associated with a timer.

[0090] In some example embodiments, the time requirement is associated with a timer, and the apparatus also includes a component for starting the timer based on determining that any of the one or more associated PDUs has been successfully transmitted.

[0091] In some example embodiments, a PDU includes at least one of the following: a Service Data Unit (SDU) of the Packet Data Convergence Protocol (PDCP) layer; a PDU of the PDCP layer; an SDU of the Service Data Access Protocol (SDAP) layer; or a PDU of the SDAP layer.

[0092] In some example embodiments, the multiple data streams include at least a video data stream.

[0093] In some embodiments, the apparatus further includes components for performing other steps in some embodiments of method 200. In some embodiments, the components include at least one processor and at least one memory including computer program code, the at least one memory and the computer program code being configured to cause the apparatus to execute together with the at least one processor.

[0094] Figure 4 A simplified block diagram of a device 400 suitable for implementing some example embodiments of the present disclosure is shown. The device 400 can be provided to implement a communication device, such as... Figure 1A The terminal device 110 or network device 120 shown. As shown, device 400 includes one or more processors 410, one or more memories 420 coupled to processor 410, and one or more communication modules 440 coupled to processor 410.

[0095] Communication module 440 is used for bidirectional communication. Communication module 440 has at least one antenna to facilitate communication. The communication interface can represent any interface required for communication with other network elements.

[0096] As a non-limiting example, processor 410 can be any type suitable for a local technology network and can include one or more of the following: general-purpose computer, special-purpose computer, microprocessor, digital signal processor (DSP), and processor based on a multi-core processor architecture. Device 400 can have multiple processors, such as application-specific integrated circuit chips that are time-dependent on a clock of a synchronous main processor.

[0097] Memory 420 may include one or more non-volatile memories and one or more volatile memories. Examples of non-volatile memories include, but are not limited to, read-only memory (ROM) 424, electrically programmable read-only memory (EPROM), flash memory, hard disk, optical disc (CD), digital video disc (DVD), and other magnetic and / or optical storage devices. Examples of volatile memories include, but are not limited to, random access memory (RAM) 422 and other volatile memories that do not persist during power outages.

[0098] Computer program 430 includes computer-executable instructions that are executed by the associated processor 410. Program 430 may be stored in ROM 424. Processor 410 may perform any suitable actions and processes by loading program 430 into RAM 422.

[0099] The embodiments of this disclosure can be implemented by program 430, enabling device 400 to perform as described in the reference. Figure 2 and 3 Any process discussed in this disclosure. Embodiments of this disclosure may also be implemented by hardware or by a combination of software and hardware.

[0100] In some example embodiments, program 430 may be tangibly contained in a computer-readable medium, which may be included in device 400 (such as in memory 420) or other storage device accessible by device 400. Device 400 may load program 430 from the computer-readable medium into RAM 422 for execution. The computer-readable medium may include any type of tangible non-volatile storage device, such as ROM, EPROM, flash memory, hard disk, CD, DVD, etc.

[0101] Figure 5 A block diagram of an example of a computer-readable medium 500 according to some exemplary embodiments of the present disclosure is shown. The computer-readable medium 500 has a program 430 stored thereon. Note that although in Figure 5 The computer-readable medium 500 is depicted in the form of a CD or DVD, but the computer-readable medium 500 may be any other form suitable for carrying or storing the program 430.

[0102] Generally, the various embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. Although various aspects of the embodiments of this disclosure are illustrated and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that, as non-limiting examples, the blocks, apparatuses, systems, techniques, or methods described herein can be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0103] This disclosure also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions that execute in a device on a target real or virtual processor, such as those included in a program module, to perform the above-referenced... Figure 2The method involves... Typically, a program module includes routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of a program module can be combined or split among program modules as needed. The machine-executable instructions used for a program module can execute on a local or distributed device. In a distributed device, program modules can reside on local and remote storage media.

[0104] Program code used to perform the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a stand-alone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0105] In the context of this disclosure, computer program code or related data may be carried by any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, etc.

[0106] Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable media can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any suitable combination thereof. More specific examples of computer-readable storage media will include electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable optical disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. As used herein, the term “non-transient” is a limitation on the medium itself (i.e., tangible, not signaling), not a limitation on the persistence of data storage (e.g., RAM and ROM).

[0107] Furthermore, although the operations are described in a specific order, this should not be construed as requiring that such operations be performed in the specific order shown or sequentially, or that all the operations shown be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are contained in the foregoing discussion, these details should not be construed as limiting the scope of this disclosure, but rather as a description of features that may be specific to particular embodiments. Certain features described in the context of a single embodiment may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0108] Although this disclosure has been described in language specific to structural features and / or methodological actions, it should be understood that the disclosure as defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are disclosed as examples of implementing the claims.

Claims

1. A communication device, comprising: At least one processor; as well as At least one memory stores instructions that, when executed by the at least one processor, cause the communication device to at least: Determine the relationships between multiple data streams; as well as Based on the aforementioned association, the multiple data streams are processed.

2. The communication device according to claim 1, wherein the plurality of data streams are respectively associated with a plurality of types of services within a multimodal service.

3. The communication device of claim 1, wherein the plurality of data streams are associated with a multimodal service identifier of a multimodal service.

4. The communication device according to any one of claims 1 to 3, wherein the association between the plurality of data streams includes one of the following: The association between multiple Quality of Service (QoS) streams mapped from the multiple data streams; or The association between multiple data radio bearers (DRBs) mapped from the multiple QoS streams of the multiple data streams.

5. The communication device according to any one of claims 1 to 4, wherein determining the association between the plurality of data streams comprises: The association between the multiple data streams is determined based on the arrival time of one or more Protocol Data Units (PDUs) of the multiple data streams.

6. The communication device according to any one of claims 1 to 5, wherein the plurality of data streams includes at least a first data stream and a second data stream, and determining the association between the plurality of data streams comprises: A time window is determined based on the margin and the arrival time of the PDUs in the first data stream; A super PDU set is determined based on the time window. The super PDU set includes one or more PDUs of the first data stream whose arrival time is within the time window, and one or more PDUs of the second data stream whose arrival time is within the time window.

7. The communication device according to any one of claims 1 to 5, wherein the plurality of data streams at least includes a first data stream and a second data stream, and determining the association between the plurality of data streams comprises: Determine a time window with a width, wherein the time windows are continuous and do not overlap; A super PDU set is determined based on the time window, the super PDU set including at least one of the following: one or more PDUs of the first data stream whose arrival time is within the time window, or one or more PDUs of the second data stream whose arrival time is within the time window.

8. The communication device according to any one of claims 1 to 5, wherein determining the association between the plurality of data streams comprises: Determine a super PDU set, which includes one or more associated PDUs across the multiple data streams.

9. The communication device according to any one of claims 1 to 8, wherein processing the plurality of data streams comprises at least one of the following: Processing logical channel priority for the multiple data streams; or Process the PDU dropping for the multiple data streams.

10. The communication device of claim 9, wherein the processing of the plurality of data streams based on the association is further based on a time requirement.

11. The communication device of claim 10, wherein the time requirement is determined based on a synchronization threshold for a multimodal service.

12. The communication device according to claim 10, wherein, The plurality of data streams includes at least a third data stream and a fourth data stream, wherein the PDU of one or more associated PDUs of the plurality of data streams that is first successfully transmitted to another communication device is the PDU of the third data stream, and the time requirement is determined based on the delay requirement of the fourth data stream compared to the third data stream.

13. The communication device according to any one of claims 10 to 12, wherein the time requirement is transmitted to another communication device.

14. The communication device of claim 13, wherein the plurality of data streams are processed by the other communication device based on the received time requirement.

15. The communication device according to claim 13 or 14, wherein the communication device is a terminal device, and the other communication device is a network device, and the communication device is further configured to: A request for uplink grant is transmitted to the network device, the uplink grant being used to transmit a super PDU set, the super PDU set comprising one or more associated PDUs of the plurality of data streams, the request including the sequence number (SN) of the super PDU set; and The uplink authorization is received from the network device, and the uplink authorization is determined based on the time requirement.

16. The communication device according to claims 1 to 15, wherein processing the plurality of data streams based on the association further comprises: When the time requirement is met, one or more of the associated PDUs of the multiple data streams are transmitted to another communication device.

17. The communication device of claim 16, wherein processing the plurality of data streams based on the association further comprises: After the time requirement is not met, one or more remaining PDUs from the one or more associated PDUs of the multiple data streams are discarded.

18. The communication device of claim 16 or 17, wherein the one or more associated PDUs are one or more PDUs in a super PDU set determined when determining the association between the plurality of data streams.

19. The communication device according to any one of claims 10 to 18, wherein the time requirement is associated with a timer.

20. The communication device according to any one of claims 16 to 18, wherein the time requirement is associated with a timer, and the communication device is further configured such that: The timer is started based on the determination that any of the one or more associated PDUs has been successfully transmitted.

21. The communication device according to any one of claims 5 to 20, wherein the PDU comprises at least one of the following: Service Data Unit (SDU) of the Packet Data Convergence Protocol (PDCP) layer; The PDU of the PDCP layer; Service Data Access Protocol (SDAP) layer SDU; or The PDU of the SDAP layer.

22. The communication device according to any one of claims 1 to 21, wherein the plurality of data streams includes at least a video data stream.

23. A method comprising: Determine the relationships between multiple data streams at the communication device; as well as The multiple data streams are processed at the communication device based on the association.

24. An apparatus comprising: A component used to determine the association between multiple data streams at a communication device; as well as A component for processing the plurality of data streams based on the association at the communication device.

25. A non-transitory computer-readable medium comprising program instructions for causing a device to perform at least the method according to claim 23.