METHOD AND DEVICES FOR SELECTING A FIRST BASE STATION OR A SECOND BASE STATION FOR SENDING A PACKAGE DATA UNIT TO A USER DEVICE (UE)

DE602016090304T2Active Publication Date: 2024-11-20NOKIA SOLUTIONS & NETWORKS OY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
DE602016090304
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Priority Date
2015-07-08
Filing Date
2016-07-07
Publication Date
2024-11-20
Estimated Expiration
2036-07-07

AI Technical Summary

Technical Problem

In data networks, particularly in Dual Connectivity scenarios, routing functions face challenges in deciding the optimal path for data packets to minimize reordering delays and optimize bandwidth usage across multiple paths, especially in non-ideal backhaul conditions.

Method used

A method that estimates the data packet path time from both base stations to the user equipment and selects the fastest path for transmitting packet data units, using algorithms like Little's Law to calculate sojourn times and update delay estimates, ensuring that packets are sent via the path with the lowest latency, thereby equalizing delays and optimizing bandwidth usage.

Benefits of technology

This approach minimizes reordering delays and Round Trip Time for TCP applications, optimizes bandwidth usage on both paths, and avoids issues like packet discards and queuing delays, providing superior performance compared to traditional routing methods.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader
Need to check novelty before this filing date? Find Prior Art

Description

BACKGROUNDTechnical Field

[0001] The exemplary and non-limiting embodiments relate generally to radio communications and, more particularly, to splitting a data stream into sub-streams.Brief Description of Prior Developments

[0002] In data networks, packets of a data stream may reach their destination via multiple paths. A Dual Connectivity operation mode is introduced in 3GPP release 12. A routing function at a "splitting point" has to decide which packets shall take which path. E-UTRAN supporting Dual Connectivity (DC) operation provides an example of such a network: A multiple RX / TX UE in RRC_CONNECTED may be configured to utilize radio resources provided by two distinct schedulers, located in two eNBs connected via a non-ideal backhaul over the X2 interface.

[0003] Further prior art can be found in document WO 2015 / 098951 A1, disclosing a mobile communication system, a control apparatus, a base station, and a user terminal. A master base station according to this document comprises a controller configured to notify a secondary base station of information for indicating whether a user plane architecture in dual connectivity is an architecture, in which a Secondary Cell Group (SCG) bearer is used, or is an architecture, in which a split bearer is used. The SCG bearer is a bearer which goes via both a serving gate way and the secondary base station without going via the master base station. The split bearer is a bearer which goes via both the serving gate way and the master base station, and which is split in a Packet Data Convergence Protocol layer of the master base station. One split path of the split bearer goes via the secondary base station, and another split path of the split bearer does not go via the secondary base station.SUMMARY

[0004] The following summary is merely intended to be exemplary. The summary is not intended to limit the scope of the claims.

[0005] In accordance with one aspect, an example method comprises determining an estimate of a data packet path time from a first base station to a user equipment (UE); determining, based at least partially upon a communication received from a second base station, an estimate of a data packet path time from the first base station via the second base station to the user equipment (UE); and based at least partially upon the determined estimates, selecting either the first base station or the second base station to transmit a packet data unit (DPU) (PDU) to the user equipment (UE).

[0006] In accordance with another aspect, an example embodiment is provided in a non-transitory program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine for performing operations, the operations comprising: determining an estimate of a data packet path time from a first base station to a user equipment (UE); determining, based at least partially upon a communication received from a second base station, an estimate of a data packet path time from the first base station via the second base station to the user equipment (UE); and based at least partially upon the determined estimates, selecting either the first base station or the second base station to transmit a packet data unit (PDU) to the user equipment (UE).

[0007] In accordance with another aspect, an example apparatus comprises means for determining an estimate of a data packet path time from a first base station to a user equipment (UE); means for determining, based at least partially upon a communication received from a second base station, an estimate of a data packet path time from the first base station via the second base station to the user equipment (UE); and means for selecting, based at least partially upon the determined estimates, either the first base station or the second base station to transmit a packet data unit (PDU) to the user equipment (UE).BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The foregoing aspects and other features are explained in the following description, taken in connection with the accompanying drawings, wherein: Fig. 1 is a diagram illustrating an example of an overall architecture of a E-UTRAN (evolved UMTS Terrestrial Radio Access) system (an air interface of 3GPP's Long Term Evolution (LTE) upgrade path for mobile networks); Fig. 2 is a diagram illustrating an example of a User Equipment (UE) in partially overlapping cells; Fig. 3 is an example diagram illustrating downlink user data flow for a split Data Radio Bearer; Fig. 4 is an example diagram illustrating downlink user data flow for a split Data Radio Bearer between MeNB and UE; Fig. 5 is an example diagram illustrating delay for the case of taking into account the data that was already sent to SeNB, but is not reflected in the received DQ_SeNB Fig. 6 is an example diagram illustrating A method for estimating the SeNB queuing delay on the MeNB side; Fig. 7 is a diagram illustrating comparing arrival times at UE using sojourn times W S and W M observable in MeNB; Fig. 8 is a diagram illustrating some components of the wireless system shown in Figs. 1 and 2; Fig. 9 is a diagram illustrating an example method; Fig. 10 is a diagram illustrating an example method. DETAILED DESCRIPTION OF EMBODIMENTS

[0009] The following abbreviations that may be found in the specification and / or the drawing figures are defined as follows: 3GPPThird Generation Partnership Program APAccess Point BBBaseband CCComponent Carrier CCAClear Channel Assignment CRCCyclic Redundancy Check CRSCell Specific Reference Signal CRWChannel Reservation Window CSIChannel State Information CSI-RSChannel State Information Reference Signal CSSCommon Search Space DCIDownlink Control Information DLDownlink DMRSDemodulation Reference Signal DRBData Radio Bearer DSDiscovery Signal DTXDiscontinuous Transmission DwPTSDownlink Pilot Time Slot e.i.r.p.equivalent isotropically radiated power elMTAEnhanced Interference Mitigation and Traffic Adaptation (the name of the 3GPP WI targeting to flexible UL / DL adaptation for TD-LTE) eNB / eNodeBenhanced Node B (base station according to LTE terminology) EPCEnhanced Packet Core EPDCCHEnhanced PDCCH FDDFrequency Division Duplex GPGuard Period IDIdentity ISMIndustrial, Scientific and Medical LAALicense-Assisted Access LBTListen Before Talk LTELong Term Evolution MeNBMaster eNB NCTNew Carrier Type OFDMOrthogonal Frequency Division Multiplexing OFDMAOrthogonal Frequency Division Multiple Access PCellPrimary Cell PDCCHPhysical Downlink Control CHannel PDCPPacket Data Convergence Protocol PDSCHPhysical Downlink Shared CHannel PDUPacket Data Unit, Protocol Data Unit PLMNPublic Land Mobile Network PRBPhysical Resource Block PSSPrimary Synchronization Signal RANRadio Access Network RelRelease RLCRadio Link Control RNTIRadio Network Temporary Identifier RRMRadio Resource Management S-GWServing Gateway SCellSecondary Cell SCSShort Control Signalling SeNBSecondary eNB SSSSecondary Synchronization Signal SDLSupplemental DL SDUService Data Unit TBTransport Block TCPTransmission Control Protocol TD / TDDTime Division duplex TLThreshold Level UEUser Equipment ULUplink UpPTSUplink Pilot Time Slot X2X2 is an interface used to communication between eNBs

[0010] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. All of the embodiments described in this Detailed Description are exemplary embodiments provided to enable persons skilled in the art to make or use the invention and not to limit the scope of the invention which is defined by the claims.

[0011] The exemplary embodiments herein describe techniques for data stream splitting. Additional description of these techniques is presented after a system into which the exemplary embodiments may be used is described.

[0012] Fig. 1 shows an example of overall architecture of an E-UTRAN system. The E-UTRAN system includes eNBs, providing an E-UTRAN user plane (PDCP / RLC / MAC / PHY) and control plane (RRC) protocol terminations towards the UE (not shown in Fig. 1). The eNBs are interconnected with each other by means of an X2 interface. The eNBs are also connected by means of a S1 interface to an EPC (Enhanced Packet Core), more specifically to a MME (Mobility Management Entity) by means of a S1 MME interface and to a Serving Gateway (S-GW) by means of a S1 interface. The S1 interface supports a many-to-many relationship between MMEs / S-GW and eNBs.

[0013] Referring also to Fig. 2, a UE 10 may be connected to more than one cell at a same time. In this example the UE 10 is connected to a first cell 12 having a base station 13 (such as an eNB for example) and a second cell 14 having a base station 15 (such as an eNB or WiFi Access Point for example). The two cells 12, 14 are, thus, at least partially overlapping. In one type of example embodiment, the first cell may operate on a licensed band and the second on may operate on an unlicensed band. For simplicity, there are just two cells depicted in the scenario shown in Fig. 2. In other alternate examples any number of cells operating on licensed and / or unlicensed band(s) may be provided to work together for a suitable Carrier Aggregation (CA).

[0014] In general, the various embodiments of the UE 10 can include, but are not limited to, cellular telephones, personal digital assistants (PDAs) having wireless communication capabilities, portable computers having wireless communication capabilities, image capture devices such as digital cameras having wireless communication capabilities, gaming devices having wireless communication capabilities, music storage and playback appliances having wireless communication capabilities, Internet appliances permitting wireless Internet access and browsing, as well as portable units or terminals that incorporate combinations of such functions.

[0015] In data networks, packets of a data stream may reach their destination via multiple paths. A routing function at a "splitting point" has to decide which packets shall take which path. Features as described herein may use a method which can be used by the splitting point in order to decide how the data stream shall be split. An algorithm may be provided which has the aim to ensure that a receiver at the destination can deliver "reordered data" as fast as possible to an application using the data. E-UTRAN supporting Dual Connectivity (DC) operation provides an example of such a network: A multiple RX / TX UE in RRC_CONNECTED is configured to utilize radio resources provided by two distinct schedulers, located in two eNBs connected via a non-ideal backhaul over the X2 interface (see 3GPP TR 36.300).

[0016] For the transport of user plane data from the S-GW to the UE so-called "split bearers" may be used. Split bearers provide two paths for downlink user plane data. They can either be sent from the S-GW via a "Master eNB (MeNB)" to the UE, or they can be sent from the S-GW via the MeNB to a Secondary eNB (SeNB) which finally sends them to the UE. For a "split bearer" the Master eNB (MeNB) is U-plane connected to the S-GW via S1-U and in addition, the MeNB is interconnected to a Secondary eNB (SeNB) via X2-U. This is illustrated by the example shown in Fig. 3.

[0017] The routing function in the PDCP layer of the MeNB decides whether a PDCP layer PDU of a split bearer is sent directly over the local air interface to the UE or whether it is forwarded to the SeNB via X2-U. A PDCP layer reordering function in the UE receives PDUs from the MeNB and from the SeNB, reorders them and forwards them to the application running on the UE 10. The features described herein may be used for splitting the data stream in such a way that PDUs sent via the two different paths experience the same or similar delay between the splitting point in the MeNB and the reordering function in the PDCP layer of the UE. Such a split i. minimizes reordering delays in the UE and the Round Trip Time for TCP ii. optimizes the usage of the available bandwidth on both paths.

[0018] The algorithm ensures that problems for TCP applications as described below cannot happen.

[0019] The PDCP layer in the MeNB stores DL SDUs of a split bearer in a bearer specific FIFO queue. When the PDCP layer processes an SDU (ciphering, adding a PDCP sequence number,..), it checks (using the algorithm of the invention) whether the direct path to the UE via the local air interface or whether the path via the air interface of the SeNB is faster. It forwards the PDU via the fastest path (see Fig. 4). The routing function in the PDCP layer of the MeNB may select, for each PDCP PDU, the fastest path from both of the available paths. This equalizes the delays via both paths, minimizes reordering delay in the UE and RTT seen by TCP. This principle can be applied not only for TCP traffic but in general. This principle does not put any specific constraints to the schedulers. QoS or fairness topics can be handled independently.

[0020] Routing of PDCP PDUs towards SeNB RLC or MeNB RLC may be done as late as possible. TRS (scheduler) and RLC queues may be kept short in order to avoid a) that packets are discarded after adding PDCP sequence numbers (after such packet discards, reordering function in UE will stop until reordering timer expires) b) that a lot of data is stuck in RLC, when the air interface throughput suddenly drops, e.g. since a UE with higher scheduling weight gets active, c) that the information that a packet has been discarded by Active Queue Management (AQM) reaches the TCP source only after a long delay (AQM should discard packets only from the SDU queue, not from the RLC queues).

[0021] Routing of PDCP PDUs towards SeNB RLC or MeNB RLC may be done early enough in order to avoid that the RLC queue of the SeNB runs empty, although the path via SeNB is faster than via MeNB and data are waiting in the SDU queue.

[0022] For the example using a Little's Law algorithm noted below, it may not be recommended that the SeNB requests packets, in order to avoid that the RLC queue runs empty. This might introduce queuing which can be avoided. For the example using SeNB feedback noted below may not use this recommendation.

[0023] The time needed to reach the UE via a given path depends on the latency of the path and the amount of queuing along the path. Queuing occurs if the arrival rate of the traffic offered to a path exceeds its capacity. The capacity of the two paths may vary quickly. For example, the capacity of the two paths may vary quickly depending on the radio conditions and the load on the air interface. The capacity of the path via the SeNB may be limited by the bandwidth available at the air interface, but it can also be limited by a bottleneck elsewhere, e. g by a bottleneck in the transport network.

[0024] The features are described below in regard to two example alternative embodiments; how to decide, whether the path to the UE via the local air interface or via the SeNB is faster. The basic principle in both of the embodiments is the same: For each (PDCP) PDU determine which of the alternative paths (transmit locally or via SeNB) is estimated to be faster. If the estimated delay of the faster path is below a target delay (e.g. 50 ms), the PDU is sent via that path, otherwise it remains still at PDCP (in MeNB). After each decision and subsequent sending of the PDU, the delay estimate of the corresponding path is updated taking account the sent PDU before making the decision for the next PDU.Algorithm using Little's Law

[0025] This example embodiment uses an algorithm based on Little's law in order to estimate, how long a packet needs to travel from the splitting function in the MeNB to the reordering function in the UE. According to Little's law the mean sojourn time W of a packet in an arbitrary stable queuing system can be calculated as W = L / TH where L is the mean number of packets in the system and TH is the mean departure rate or, in other words, the mean throughput of the system.Decision Criterion

[0026] The MeNB has to decide for each PDCP layer PDU which of the two paths shown in Figure 4 shall be used. The MeNB may use the following information for calculating the sojourn times of packets using one of the two paths shown in Figure 4: 1) The SeNB sends messages ("acknowledgement (ack) messages") to the MeNB which indicate that PDUs have been sent to the UE (or, depending on the definition, that PDUs have been correctly received by the UE). There are several options how to exactly define what "sent to the UE" means. "Sent to the UE" can be defined as: The SW component of the SeNB creating ack messages is informed by the RLC layer that the PDU has been successfully delivered to the UE. Alternatively "Sent to the UE" can, for example, be defined as: The SW component of the SeNB creating ack messages is informed by the RLC that the RLC has transferred the PDU from RLC to MAC. Another alternative is messaging related to Downlink Data Delivery Status procedure: 3GPP TS 36.425 defines DL data delivery status messages for forwarding acks for PDUs from SeNB to MeNB, which includes the highest PDCP PDU sequence number successfully delivered in sequence to the UE among those PDCP PDUs received from the MeNB. 2) MeNB maintains a variable L_S ("data in flight"), which stores the number of bytes included in PDUs which have been routed to the SeNB, but which have not yet been acknowledged by the SeNB. 3) MeNB maintains a variable TH_S, which provides an estimation for the "expected throughput" of the path via SeNB in the near future. This means it is assumed that the PDUs currently queued will be served at TH_S. The MeNB can estimate TH_S by calculating the "recent throughput" from acknowledgements for PDUs contained in the ack messages sent by the SeNB. During the starting phase the "recent throughput" may be a poor estimation for the "expected throughput". Therefore it is recommended to estimate the "expected throughput" TH_S as max(lnitial_throughput, recent throughput), where "lnitial_throughput" is a predefined value. For example, the predefined value may be set to 1 Mb / s. 4) MeNB maintains a variable L_M ("data in flight"), which contains the number of bytes of PDUs in flight. A PDU is in flight, when the PDCP layer has taken the decision to send the PDU via the local air interface and is waiting for the information that the PDU has been sent to the UE (applying an analogous definition of "sent to UE" as in the SeNB). 5) MeNB maintains a variable TH_M which gives the "expected throughput" of the path via MeNB. TH_M is to be provided for example by the MAC scheduler.

[0027] Let two denote the time when the PDCP layer of the MeNB decides to route a PDU via the local air interface. Let t M1 denote the time when the PDCP layer is informed that the PDU has been sent to the UE T_MeNB = L_M + PDU-size / TH_M is an estimate for the sojourn time t M1 - t M0 for the PDU currently processed by the PDCP layer.

[0028] Let t S0 denote the time when the PDCP layer of the MeNB decides to route a PDU via SeNB. Let t S1 denote the time when the SW component of the SeNB creating ack messages is informed that the PDU has been sent to the UE, and let t S2 denote the time when the PDCP layer in the MeNB receives the ack message confirming that the PDU has been sent to the UE. L_S + PDU-size / TH_S is an estimate for the sojourn time t S2 - tso for the PDU currently processed by the PDCP layer. Let T_ack denote the time difference t S2 - t S1 . Then L_S + PDU − size / TH_S − T_ack is an estimate for (t S2 - t S0 ) - (t S2 - t S1 ) = t S1 - t S0 for the PDU currently processed by the PDCP layer.

[0029] Let T_ack_avg denote the average of the values of T_ack. T_ack_avg includes the latency of the path from SeNB to MeNB, the mean delay between "sending a PDU to the UE" (as defined above) and sending out the ack message to the MeNB, and processing times in SeNB and MeNB. T_ack_avg may be calculated offline for a given pair of eNBs. L_S + PDU-size / TH_S − T_ack_avg gives an estimation of the time t S1 - t S0 which can be calculated by the MeNB.

[0030] Comparing the values given by (2) and (5) the MeNB can decide which of the two paths is faster.

[0031] Simulations have shown that the estimate (5) is working well, when data transfer is ongoing. However, modifications may be needed for the starting phase, when the path via SeNB has not yet been used and thus L_S=0 and TH_S = "Initial_Throughput". In such a situation (5) would give a negative value.

[0032] For the starting phase (i.e. the PDUs counted within L_S have not yet been sent to the UE) an estimate for the time which a PDU needs to reach the UE via SeNB can be calculated by X2_latency + L_S + PDU-size / TH_S

[0033] Taking into account (6) the time which a PDU needs to reach the UE via SeNB (or more precisely the time tS1 - tS0) can be estimated by the MeNB in the following way. Define

[0034] There are various ways how to define the starting phase. In the simplest case the length of the starting phase is set to 0, this means (7) is never used. As an example of an alternatively, one can start a timer T_starting_phase when the variable L_S is incremented from 0 to a value >0 and define "data transfer is in the starting phase" if and only if T_starting_phase <_ X2_latency.

[0035] Taking into account (2), (7) and (8) one can now define the decision criterion: A PDU should be routed via SeNB if and only if T_SeNB < T_MeNB

[0036] One can use the "Initial_Throughput" used in the definition of TH_S to tune the allowed delay, before the path via SeNB is taken into use. Optimal values for "Inital_Throughput" depend on the definition of the starting phase. For example, for a starting phase of length 0, an X2-delay of 5 ms, a PDU size of 1300 bytes and a Minimum_Throughput of 1 Mb / s, the path via SeNB is used for the first time when a PDU would need via MeNB more than X2_latency + (PDU-size / TH_S) = 15,4 ms. For a Minimuim_Throughput of 500 kbit / s it would be used for the first time, when a PDU would need via MeNB more than 25,8 ms. One alternative for setting the "Initial_Throughput" value is to base it at least partially on UE's SeNB link quality, which can be known at MeNB based on UE's measurement reports such as the Reference Signal Received Quality (RSRQ), so that better RSRQ link quality translates into higher "Initial_Throughput".

[0037] Fig. 7 illustrates comparison arrival times at a UE using sojourn times W S and W M observable in MeNB. For the system for calculating the sojourn time Ws, a segment of a PDCP layer PDU (in short PDU) enters the "system", when the MeNB decides to forward the PDU to the SeNB. A PDU leaves the system, when a DL Data Delivery Status message (or ack message) arrives at the MeNB indicating that the PDCP PDU (or a PDCP PDU with higher sequence number) has been successfully delivered in sequence to the UE.. L S is the sum of PDU sizes of the "PDUs in the system" ("data in flight"). Th S , mean departure rate (throughput). For the system for Calculating the sojourn time W M , a PDU enters the "system" when it is inserted into the RLC queue in the MeNB. It leaves the system when RLC forwards the last segment of the PDU or the last segment of a PDU with higher PDCP sequence number to MAC. L M is the sum of PDU sizes of the "PDUs in the system" ("data in flight"). Th M mean departure rate (throughput).

[0038] (9) stated a different way may have a criterion for PDCP PDU Data Routing (Split between MeNb and SeNB) which may include: Path via MeNB faster than path via SeNB, if W M < W S − T_ack_avg where T_ack_avg as defined above.

[0039] According to Little's Law W M = L M / Th M and Ws = Ls / Ths. It follows that (10) is equivalent with L M / Th M < L S / Th S − T_ack_avg

[0040] The SeNB throughput estimate Th S can be calculated in the MeNB for example as Sum of PDU sizes of PDUs passed to MAC / Sending Period (i.e. T_DS)

[0041] The MeNB throughput estimate Th M can be calculated in an analogous way, alternatively for example a value provided directly from the scheduler can be used.Late forwarding of PDUs

[0042] The PDCP layer of the MeNB should forward SDUs to one of the two available paths as late as possible in order to keep the amount of PDUs between the splitting point in the MeNB and the reordering function in the UE small. This helps to avoid the following problems: i. If a lot of data is stored in the SeNB and the available capacity at the air interface is suddenly decreasing high reordering delays will occur in the UE. For example, the available capacity at the air interface may suddenly decrease due to changing radio conditions. ii. If queues between PDCP layer routing function in the MeNB and reordering function in the UE (this means after adding the PDCP sequence number) overflow and PDUs are lost, the reordering function in the UE will stop forwarding PDUs to the next layer until the reordering timer expires. This interruption may take several hundreds of milliseconds. iii. The Active Queue Management (AQM) of an eNB may discard packets in order to trigger TCP congestion control. Due to the reordering problem described under ii) AQM should discard packets only from the SDU queue and not from the RLC queues in MeNB or SeNB (see Figure 4). If there are a lot of packets waiting in the RLC queues then a packet discard from the SDU queue will be noted by the TCP source only after a long delay.

[0043] On the other hand there is also a risk that SDUs are forwarded too late. It should not happen that SDUs are queued in the MeNB while the RLC queue in the SeNB runs empty and available capacity at the air interface of the SeNB is not used.

[0044] The proposed algorithm removes an SDU from the PDCP queue, only if the expected delay to the UE is below a given "target delay", such as 50 ms for example. Together with inequality (9) this constraint leads to the following algorithm.PDCP Routing Algorithm

[0045] Step 1: Use inequality (9) in order to decide whether the SDU at the head of the PDCP layer queue should be sent via SeNB or via MeNB.

[0046] Step 2: Check whether the estimated delay to the UE for the faster path is smaller or equal to the target delay (e.g. using the estimates (2), respectively (7 or 8)).

[0047] Step 3: If the check under Step 2 gives a positive result, then process the SDU and forward the resulting PDU towards the faster path. Otherwise keep the SDU in the PDCP layer queue.Algorithm using feedback from SeNB

[0048] Features as described herein may use the delay estimates for both MeNB and SeNB, which take into account the X2 delay (for SeNB), and the amount of data that is already in the queue (i.e. previous packets that went to MeNB / SeNB). As a result, if a large chunk of PDUs is routed via SeNB, those should typically not be consecutive. For example, the PDUs routed via SeNB could be every second packet, or 2 out of 3 so that the amount of reordering needed at UE PDCP is reduced. In the end, PDCP reordering no longer has a big effect, because there is a higher probability that the PDUs come in order via the MeNB and SeNB paths.

[0049] One way to implement this is that SeNB sends to MeNB the following information for each UE it is serving and that is dual connected to MeNB (such as periodically for example) 1) Queuing delay, denoted here as DQ_SeNB: How long it is estimated to take to transmit the pending data at the SeNB 2) Estimated SeNB throughput, denoted as TP_SeNB (e.g. past throughput, or for new data e.g. in the next ~T ms)

[0050] Additionally it is assumed that MeNB knows (can estimate) the X2 delay, denoted here as D_X2. This could be done outside or separately from the flow control.

[0051] In this alternative embodiment, the flow control on the MeNB side would run with the following algorithm for example at regular intervals (e.g. once per millisecond or subframe), or when there is a data request from SeNB, or when there is / arrives data in the PDCP buffer and it is to be determined where it is routed (via MeNB or via SeNB)

[0052] :

[0053] The way the delay impacts the knowledge on the data availability on the MeNB side, it is illustrated in 5. Figure 5 is a delay diagram for the case of taking into account the data that was already sent to SeNB, but is not reflected in the received DQ_SeNB, i.e. during last 2*D_X2.

[0054] One other alternative implementation of the algorithm would work with: Sending throughput estimates of SeNB to MeNB over X2 interface ∘ Periodicity could be configurable, or this information is sent together with PDCP status reports (such as a part of Downlink Data Delivery Status procedure, or other signalling between SeNB and MeNB). ∘ In an alternative embodiment the UE throughput on SeNB could be also estimated based on MeNB observing the received SeNB indications that the PDUs were successfully delivered to UE (i.e. the throughput at PDCP level). SeNB queuing delay may be more difficult to send over X2. ∘ If the SeNB can send the queuing delay information to MeNB, the SeNB can estimate the queuing delay based on dividing the amount of pending data for the UE in SeNB with the estimate UE throughput. The pending data could be taken from RLC level (data that is not yet acknowledged in RLC status report) or MAC level (HARQ transmission still pending). The former option is more accurate, but slower. The latter alternative is faster (as HARQ feedback is fast), but may suffer from occasional errors due to NACK->ACK misinterpretation. ∘ MeNB could estimate the SeNB queuing delay based on: a) SeNB request for data e.g. as part of Downlink Data Delivery Status procedure indicating desired buffer size for UE or E-RAB. ( E.g. a large buffer size request could indicate a low level of data in SeNB buffer) and b) Estimate the queuing delay DQ_SeNB = SeNB_T_buffering - SeNB_data_request / SeNB_throughput_estimate, where SeNB_T_buffering is the target buffering time at SeNB or target delay (this could be for example configured by MeNB or negotiated between MeNB and SeNB earlier), SeNB_data_request is the size of desired buffer size indicated by the SeNB, SeNB_throughput_estimate is the SeNB throughput for the UE which could be e.g. signalled from SeNB to MeNB or estimated by MeNB from PDCP status reports. See Figure 6. ∘ In an alternative embodiment the SeNB queuing delay could be estimated also based on MeNB observing the difference between 1) PDU was routed from MeNB to SeNB, and 2) MeNB receives from SeNB an indication that the PDU was successfully delivered to UE. The estimated time it takes to receive the acknowledgement / message from UE->SeNB->MeNB could be subtracted from this estimate (this includes X2 delay and UL scheduling delay, as well as possibly some processing times).

[0055] In addition, it is assumed that MeNB knows (can estimate) the X2 delays.

[0056] Figure 6 illustrates a method for estimating the SeNB queuing delay on the MeNB side. MeNB configures the target buffering time of the SeNB T_buffering, SeNB send the data request to MeNB, MeNB estimates the queuing delay as DQ_SeNB = T_buffering - data_request_size / SeNB_throughput_estimate. The assumption here is that the SeNB requests data until the expected / estimated delay for new data is T_buffering. I.e. the SeNB targets to have for the UE such amount of data in the buffer that the time taken to serve all of it is equal to T_buffering.

[0057] Features as described above may be used to minimize reordering delays in the UE and the Round Trip Time for TCP, and optimize the usage of the available bandwidth on both paths. Simulation for the Little's Law algorithm described above have shown that Dual Connectivity with this algorithm provides similar performance as Intra eNB Carrier Aggregation. Simulation results have also confirmed that the SeNB data request algorithm described above avoids unnecessary reordering and, thus, gives superior performance compared to responding to SeNB data requests without comparing the delay through SeNB versus delay through MeNB.

[0058] The Little's Law algorithm described above may take into account congestion in the transport function of the MeNB. For example, this may be caused by high processing load for Internet Protocol Security (IPSec). In case of transport congestion inside the MeNB, the algorithm will notice growing delays and automatically reduce the transport load by reducing the traffic forwarded to the SeNB. In this way complex MeNB internal overload protection mechanisms like backpressure from Transport to U-Plane can be avoided.

[0059] The algorithm takes into account congestion in the transport network (in the same way as MeNB internal congestion is taken into account). Thus, very expensive over-dimensioning of the X2 links between MeNB and SeNB can be avoided which would be required by other algorithms.

[0060] For the SeNB data request algorithm described above, the X2 delay (if highly varying) may need to be estimated outside the algorithm. For this purpose the MeNB could perform this estimation (it has other uses as well) for example, or DC messaging could include time stamps which would be also useful for discarding packets at SeNB for example. Rough estimates could be also made based on network topology.

[0061] Referring also to Fig. 8, in the wireless system 230 a wireless network 235 is adapted for communication over a wireless link 232 with an apparatus, such as a mobile communication device which may be referred to as a UE 10, via a network access node, such as a Node B (base station), and more specifically an eNB 13. The network 235 may include a network control element (NCE) 240 that may include MME / S-GW functionality, and which provides connectivity with a network, such as a telephone network and / or a data communications network (e.g., the internet 238).

[0062] The UE 10 includes a controller, such as a computer or a data processor (DP) 214, a computer-readable memory medium embodied as a memory (MEM) 216 that stores a program of computer instructions (PROG) 218, and a suitable wireless interface, such as radio frequency (RF) transceiver 212, for bidirectional wireless communications with the eNB 13 via one or more antennas.

[0063] The eNB 13 also includes a controller, such as a computer or a data processor (DP) 224, a computer-readable memory medium embodied as a memory (MEM) 226 that stores a program of computer instructions (PROG) 228, and a suitable wireless interface, such as RF transceiver 222, for communication with the UE 10 via one or more antennas. The eNB 13 is coupled via a data / control path 234 to the NCE 240. The path 234 may be implemented as an interface. The eNB 13 may also be coupled to another eNB via data / control path 236, which may be implemented as an interface.

[0064] The NCE 240 includes a controller, such as a computer or a data processor (DP) 244, a computer-readable memory medium embodied as a memory (MEM) 246 that stores a program of computer instructions (PROG) 248.

[0065] At least one of the PROGs 218, 228 and 248 is assumed to include program instructions that, when executed by the associated DP, enable the device to operate in accordance with exemplary embodiments of this invention, as will be discussed below in greater detail. That is, various exemplary embodiments of this invention may be implemented at least in part by computer software executable by the DP 214 of the UE 10; by the DP 224 of the eNB 13; and / or by the DP 244 of the NCE 240, or by hardware, or by a combination of software and hardware (and firmware). Base station 15 may have the same type of components as the base station 13.

[0066] For the purposes of describing various exemplary embodiments in accordance with this invention the UE 10 and the eNB 13 may also include dedicated processors, for example RRC module 215 and a corresponding RRC module 225. RRC module 215 and RRC module 225 may be constructed so as to operate in accordance with various exemplary embodiments in accordance with this invention.

[0067] The computer readable MEMs 216, 226 and 246 may be of any type suitable to the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor based memory devices, flash memory, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. The DPs 214, 224 and 244 may be of any type suitable to the local technical environment, and may include one or more of general purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs) and processors based on a multicore processor architecture, as non-limiting examples. The wireless interfaces (e.g., RF transceivers 212 and 222) may be of any type suitable to the local technical environment and may be implemented using any suitable communication technology such as individual transmitters, receivers, transceivers or a combination of such components.

[0068] According to the 3GPP specification the SeNB should request data according to a "desired buffer size" from the MeNB. The MeNB may send as much data to the SeNB as requested. "Split Data Radio bearers" or simply "split bearers" are used in order to increase the DL throughput of individual bearers compared to bearers not using Dual Connectivity. The PDCP layer routing function in the MeNB has the task to decide whether an arriving downlink SDU of a split bearer shall be forwarded to the local RLC layer, or whether it shall be forwarded via X2-U interface to the SeNB. This means, the routing function splits the arriving data stream into two sub-streams using the two different paths, the direct path and the path via the SeNB.

[0069] Features as described herein deal with the question of how to optimally split the arriving data stream. If the splitting is done in a sub-optimal way the throughput of the split bearer may be worse than without using Dual Connectivity.

[0070] Consider for example a conventional application using TCP. Assume that the capacity of the path via the SeNB is 1 Mb / s and that the capacity of the path via the MeNB is 10 Mb / s and that the DL data stream is splitted according to a ratio 50:50. Further assume for simplicity that there is no packet loss. The reordering function in the UE will deliver data to the TCP layer at a rate of 2 Mb / s. TCP will send acknowledgements for the DL data to the TCP source at a rate of 2 Mb / s. After slow start and filling the congestion window the TCP source will react with adjusting the sending rate to 2 Mb / s. This means the capacity of the path via MeNB will not be fully used. Without activating Dual Connectivity the TCP source would send at 10 Mb / s, instead of 2 Mbit / s.

[0071] It has to be noted that the throughput of the path via the SeNB may be determined by a bottleneck outside the SeNB. The bottleneck can, for example, be in the Transport SW of the MeNB feeding the X2 interface. This may happen in particular if IPSec is used on the X2 interface and a lot of CPU power is needed for ciphering packets sent via X2. A bottleneck may also occur in a router in the path between MeNB and SeNB. It is important that the splitting algorithm running in the PDCP layer of the MeNB takes into account such potential bottlenecks. Otherwise problems as described in the above example may occur.

[0072] A communications system and / or a network node / base station may comprise a network node or other network elements implemented as a server, host or node operationally coupled to a remote radio head. At least some core functions may be carried out as software run in a server (which could be in the cloud) and implemented with network node functionalities in a similar fashion as much as possible (taking latency restrictions into consideration). This is called network virtualization. "Distribution of work" may be based on a division of operations to those which can be run in the cloud, and those which have to be run in the proximity for the sake of latency requirements. In macro cell / small cell networks, the "distribution of work" may also differ between a macro cell node and small cell nodes. Network virtualization may comprise the process of combining hardware and software network resources and network functionality into a single, software-based administrative entity, a virtual network. Network virtualization may involve platform virtualization, often combined with resource virtualization. Network virtualization may be categorized as either external, combining many networks, or parts of networks, into a virtual unit, or internal, providing network-like functionality to the software containers on a single system.

[0073] Referring also to Fig. 9, an example method may comprise, such as at MeNB, determining an estimate of a data packet path time from a first base station to a user equipment (UE) as illustrated by block 100; determining, based at least partially upon a communication received from a second base station, an estimate of a data packet path time from the first base station via the second base station to the user equipment (UE) as illustrated by block 102; and based at least partially upon the determined estimates, selecting either the first base station or the second base station to transmit a packet data unit (PDU) to the user equipment (UE) as illustrated by block 104.

[0074] Referring also to Fig. 10, an example method may comprise, such as at SeNB, as illustrated by block 106 determining information by a second base station comprising at least one of: a number of packet data units (PDUs) received by the second base station from a first base station which have been sent by the second base station to a user equipment (UE), a queuing delay in the second base station in transmitting the packet data units (PDUs) received by the second base station from the first base station to the user equipment (UE), and an estimate of throughput by the second base station in transmitting the packet data units (PDUs) received by the second base station from the first base station to the user equipment (UE); and

[0075] transmitting the information by the second base station to the first base station as illustrated by block 108.

[0076] Fields in DL DATA DELIVERY STATUS message (or ack message) sent from SeNB to MeNB may be used. The highest successfully delivered PDCP Sequence Number may indicate the PDCP PDU with the highest sequence number which has been successfully delivered to the UE. Copies of PDCP PDUs may be stored in MeNB for handover until this message tells, that they can be discarded. The desired buffer size for the E-RAB may be used to indicate the maximum number of bytes which can be accepted by the SeNB: Calculated as "available buffer space - margin". Desired buffer size for the UE may indicate the PDCP PDU with the highest X2-U sequence number which has been passed to MAC layer for transmission (i.e. the RLC layer has passed the last segment of the PDU to the MAC layer). All other fields may be used as described in the standard.

[0077] An Algorithm for Data Splitting Taking into Account Congestion on X2 and in eNB may be provides such as the example given below.

Claims

1. A method comprising: determining (100) an estimate of a data packet path time from a first base station (13) to a user equipment (10); determining (102), based at least partially upon a communication received from a second base station (15), an estimate of a data packet path time from the first base station via the second base station to the user equipment; and based at least partially upon the determined estimates, selecting (104) either the first base station or the second base station to transmit a packet data unit to the user equipment.

2. A method as in claim 1 where, when the estimated data packet path time via a selected one of the base stations (13, 15) is below a target delay, forwarding the packet data unit to the selected base station.

3. A method as in any one of claims 1 to 2 where, the first base station (13) selects (104) either the first base station or the second base station (15) to transmit the packet data unit to the user equipment (10), and / or selecting (104) either the first base station or the second base station to transmit the packet data unit is done early enough to avoid a radio link control queue of the second base station from running empty.

4. A method as in any one of claims 1 to 3 where, the estimate of a data packet path time is determined as a ratio based on queued packet data units and throughput of the queued packet data units, or the estimate of a data packet path time is dependent on a size of the packet data units sent to the user equipment (10), or the estimate of a data packet path time is determined based on recent throughput of data packets transmitted via the second base station (15) to the user equipment, or the estimate of a data packet path time is determined as a ratio of the mean number of queued packet data units sent to the user equipment and the mean throughput of the queued packet data units.

5. A method as in any one of claims 1 to 3 where, the estimate of a data packet path time is dependent on feedback sent from the second base station (15) to the first base station (13).

6. A method as in claim 5, wherein the feedback includes at least one of a queuing delay of the second base station (15), an estimate of throughput of the second base station and a data traffic of a data transport network that includes the second base station.

7. A method as in claim 1 further comprising, delaying forwarding of the packet data unit by the first base station (13) to a path to the user equipment (10) in order to keep a quantity of packet data units between a splitting point in the first base station and a reordering function in the user equipment low.

8. A non-transitory program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine for performing operations, the operations comprising: determining (100) an estimate of a data packet path time from a first base station (13) to a user equipment (10); determining (102), based at least partially upon a communication received from a second base station (15), an estimate of a data packet path time from the first base station via the second base station to the user equipment; and based at least partially upon the determined estimates, selecting (104) either the first base station or the second base station to transmit a packet data unit to the user equipment.

9. An apparatus comprising: means for determining an estimate of a data packet path time from a first base station (13) to a user equipment (10); means for determining, based at least partially upon a communication received from a second base station (15), an estimate of a data packet path time from the first base station via the second base station to the user equipment; and means for selecting, based at least partially upon the determined estimates, either the first base station or the second base station to transmit a packet data unit to the user equipment.

10. An apparatus as in claim 9 further comprising means for delaying forwarding of the packet data unit by the first base station (13) to a path to the user equipment (10) in order to keep a quantity of packet data units between a splitting point in the first base station and a reordering function in the user equipment low.