Method and apparatus for generating a sector carrier reassignment plan

Abstract

A method is disclosed to generate a sector carrier reassignment plan. The method includes generating a plurality of fixed clusters of sector carriers using an agglomerative clustering algorithm, a distance matrix of a plurality of sector carriers, and a current sector carrier assignment, generating, for each of the plurality of fixed clusters, a list of candidate sector carriers of the fixed cluster using the agglomerative clustering algorithm, a list of sector carriers included in the fixed cluster, and a list of sector carriers that have not yet been added to a fixed cluster, adding one or more candidate sector carriers included in the lists of candidate sector carriers to corresponding fixed clusters, adding any sector carriers that have not yet been added to a fixed cluster to one of the plurality of fixed clusters, and generating the sector carrier reassignment plan based on the plurality of fixed clusters.

Description

SPECIFICATIONMETHOD AND APPARATUS FOR GENERATING A SECTOR CARRIER REASSIGNMENT PLANTECHNICAL FIELD

[0001] Embodiments of the invention relate to the field of telecommunication networks, and more specifically, to generating a sector carrier reassignment plan in a telecommunication network.BACKGROUND

[0002] Cellular telecommunication networks, which are sometimes referred to as “mobile networks,” are relatively large networks encompassing a large number of wireless devices (sometimes referred to as “user equipment” (UE) or “mobile devices”). The mobile network is typically connected to one or more other networks (e.g., the Internet) and allow mobile devices to connect to these other networks. The mobile network is designed to allow the mobile devices (e.g., mobile phones, tablets, laptops, Internet of Things (loT) devices and similar devices), to shift connection points with the mobile network in a manner that maintains continuous connections for the applications of the mobile devices. Typically, the mobile devices connect to the mobile network via radio access network (RAN) base stations (sometimes referred to as “access points”), which provide connectivity to a number of mobile devices for a local area or “cell .” Managing and configuring the mobile network including the cells of the mobile network is an administrative challenge, as each cell can have different geographic and / or technological characteristics.

[0003] In modern implementations of mobile networks, such as Fifth Generation (5G) mobile networks or beyond-5G mobile networks, a number of different resources such as radio resources and computing resources (e g., compute, storage, and / or transport resources) are provisioned to provide communication connections and services to a number of mobile devices. Achieving suitable performance of the mobile network, such as service availability and service assurance, may require the real-time prioritization of the different provisioned resources. The prioritization may be challenging, as a finite amount of resources are balanced with a number of competing and sometimes contradicting objectives while still meeting requirements on resiliency, mobility, server utilization, migration and / or energy efficiency requirements of the mobile network. For example, in a cloud-based RAN, resources are provisioned for virtualized distributed units (DUs), centralized units providing user planefunctionality (CU-UP), centralized units providing control plane functionality (CU-CP), and the routing of enhanced Common Public Radio Interface (eCPRI) traffic to the virtualized DUs in hub sites. The cloud-based RAN may have competing objectives, such as a high mobility requirement that incentivizes hosting multiple virtualized DU instances on a single server, as well as a resiliency requirement that incentivizes hosting the multiple virtualized DU instances on different servers.

[0004] For implementations of mobile networks in 5G and beyond, efficient and robust end-to- end resource provisioning that involves radio, compute, storage, and transport, is a key challenge to ensure service availability Additionally, real-time prioritization of the computing resources to achieve a high level of performance and resilience across all domains is a must for service assurance.

[0005] When situational changes occur in the mobile network, such as when some sector carriers are removed from the mobile network and / or when there is a change in traffic volumes in the mobile network, there is a need to reassign sector carriers to adapt to the changes. Any sector carrier reassignment plan should take into consideration the requirements / preferences between sector carriers (e.g., mobility preferences and resiliency preferences), while trying to minimize the number of sector carrier migrations to reduce the overhead involved with migration. Existing approaches for generating a sector carrier reassignment plan model the problem as a linear programming problem and solve the linear programming problem using exact optimization solvers. While existing approaches can provide a globally optimal solution, they do not scale well (e.g., when there are hundreds or thousands of sector carriers).SUMMARY

[0006] An embodiment disclosed herein is a method performed by a computing device to generate a sector carrier reassignment plan in a telecommunication network. The method includes obtaining a distance matrix of a plurality of sector carriers, generating a plurality of fixed clusters of sector carriers using an agglomerative clustering algorithm, the distance matrix, and a current sector carrier assignment, generating, for each of the plurality of fixed clusters, a list of candidate sector carriers of the fixed cluster using the agglomerative clustering algorithm, a list of sector carriers included in the fixed cluster, and a list of sector carriers that have not yet been added to a fixed cluster, adding one or more candidate sector carriers included in the lists of candidate sector carriers to corresponding fixed clusters, adding any sector carriers that have not yet been added to a fixed cluster to one of the plurality of fixed clusters, and generating the sector carrier reassignment plan based on the plurality of fixed clusters.

[0007] An embodiment disclosed herein is a non-transitory machine-readable medium comprising computer program code which when executed by a computing device carries out operations for generating a sector carrier reassignment plan in a telecommunication network. The operations include obtaining a distance matrix of a plurality of sector carriers, generating a plurality of fixed clusters of sector carriers using an agglomerative clustering algorithm, the distance matrix, and a current sector carrier assignment, generating, for each of the plurality of fixed clusters, a list of candidate sector carriers of the fixed cluster using the agglomerative clustering algorithm, a list of sector carriers included in the fixed cluster, and a list of sector carriers that have not yet been added to a fixed cluster, adding one or more candidate sector carriers included in the lists of candidate sector carriers to corresponding fixed clusters, adding any sector carriers that have not yet been added to a fixed cluster to one of the plurality of fixed clusters, and generating the sector carrier reassignment plan based on the plurality of fixed clusters.

[0008] An embodiment disclosed herein is a computing device to generate a sector carrier reassignment plan in a telecommunication network. The network device includes one or more processors and a non-transitory machine-readable storage medium that stores instructions, which when executed by the one or more processors, causes the computing device to perform operations for generating a sector carrier reassignment plan in a telecommunication network. The operations include obtaining a distance matrix of a plurality of sector carriers, generating a plurality of fixed clusters of sector carriers using an agglomerative clustering algorithm, the distance matrix, and a current sector carrier assignment, generating, for each of the plurality of fixed clusters, a list of candidate sector carriers of the fixed cluster using the agglomerative clustering algorithm, a list of sector carriers included in the fixed cluster, and a list of sector carriers that have not yet been added to a fixed cluster, adding one or more candidate sector carriers included in the lists of candidate sector carriers to corresponding fixed clusters, adding any sector carriers that have not yet been added to a fixed cluster to one of the plurality of fixed clusters, and generating the sector carrier reassignment plan based on the plurality of fixed clusters.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The invention may best be understood by referring to the following description and accompanying drawings that are used to illustrate embodiments of the invention. In the drawings:

[0010] Figure 1 is a diagram showing a system that can generate a sector carrier reassignment plan, according to some embodiments.

[0011] Figure 2 is a flow diagram showing operations of phases one and two of generating a SC reassignment plan, according to some embodiments.

[0012] Figure 3 is a flow diagram showing operations of phase three of generating a SC reassignment plan, according to some embodiments.

[0013] Figure 4 is a flow diagram showing operations of phase four of generating a SC reassignment plan, according to some embodiments.

[0014] Figure 5 is a diagram showing a current SC assignment and a SC reassignment, according to some embodiments.

[0015] Figure 6 is a diagram showing a distance matrix, according to some embodiments.

[0016] Figure 7 is a diagram showing the generation of the first fixed cluster, according to some embodiments.

[0017] Figure 8 is a diagram showing generation of the second fixed cluster, according to some embodiments.

[0018] Figure 9 is a diagram showing the generation of the third fixed cluster, according to some embodiments.

[0019] Figure 10 is a diagram showing fixed clusters, according to some embodiments.

[0020] Figure 11 is a diagram showing generating candidates of the first fixed cluster, according to some embodiments.

[0021] Figure 12 is a diagram showing the list of candidates of the fixed clusters, according to some embodiments.

[0022] Figure 13 is a diagram showing the addition of a unique candidate to a fixed cluster, according to some embodiments.

[0023] Figure 14 is a diagram showing adding single candidates, according to some embodiments.

[0024] Figure 15 is a diagram showing the addition of an SC to a fixed cluster that is most similar to a cluster that includes the SC in the current SC assignment, according to some embodiments.

[0025] Figure 16 is a diagram showing the addition of an SC to a fixed cluster that is most similar to a cluster that includes the SC in an initial clustering result, according to some embodiments.

[0026] Figure 17 shows the final fixed clusters, which represent the SC reassignment plan.

[0027] Figure 18 is a diagram showing an example hierarchy of computing resources, according to some embodiments.

[0028] Figure 19 is a flow diagram of a method for generating a sector carrier reassignment plan, according to some embodiments.

[0029] Figure 20 is a flow diagram showing operations to generate fixed clusters, according to some embodiments.

[0030] Figure 21 is a flow diagram showing operations to generate lists of candidate sector carriers of fixed clusters, according to some embodiments.

[0031] Figure 22 is a flow diagram showing operations to add sector carriers included in the lists of candidate sector carriers to fixed clusters, according to some embodiments.

[0032] Figure 23 is a flow diagram showing operations to add sector carriers that have not yet been added to a fixed cluster to a fixed cluster, according to some embodiments.

[0033] Figure 24 illustrates connectivity between network devices (NDs) within an exemplary network, as well as three exemplary implementations of the NDs, according to some embodiments of the invention.DETAILED DESCRIPTION

[0034] The following description describes methods and apparatus for generating a sector carrier reassignment plan in a telecommunication network. In the following description, numerous specific details such as logic implementations, opcodes, means to specify operands, resource partitioning / sharing / duplication implementations, types and interrelationships of system components, and logic partitioning / integration choices are set forth in order to provide a more thorough understanding of the present invention. It will be appreciated, however, by one skilled in the art that the invention may be practiced without such specific details. In other instances, control structures, gate level circuits and full software instruction sequences have not been shown in detail in order not to obscure the invention. Those of ordinary skill in the art, with the included descriptions, will be able to implement appropriate functionality without undue experimentation.

[0035] References in the specification to “one embodiment,” “an embodiment,” “an example embodiment,” etc., indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to affect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.

[0036] Bracketed text and blocks with dashed borders (e.g., large dashes, small dashes, dotdash, and dots) may be used herein to illustrate optional operations that add additional features to embodiments of the invention. However, such notation should not be taken to mean that theseare the only options or optional operations, and / or that blocks with solid borders are not optional in certain embodiments of the invention.

[0037] In the following description and claims, the terms “coupled” and “connected,” along with their derivatives, may be used. It should be understood that these terms are not intended as synonyms for each other. “Coupled” is used to indicate that two or more elements, which may or may not be in direct physical or electrical contact with each other, co-operate or interact with each other. “Connected” is used to indicate the establishment of communication between two or more elements that are coupled with each other.

[0038] An electronic device stores and transmits (internally and / or with other electronic devices over a network) code (which is composed of software instructions and which is sometimes referred to as computer program code or a computer program) and / or data using machine-readable media (also called computer-readable media), such as machine-readable storage media (e.g., magnetic disks, optical disks, solid state drives, read only memory (ROM), flash memory devices, phase change memory) and machine-readable transmission media (also called a carrier) (e.g., electrical, optical, radio, acoustical or other form of propagated signals - such as carrier waves, infrared signals). Thus, an electronic device (e.g., a computer) includes hardware and software, such as a set of one or more processors (e.g., wherein a processor is a microprocessor, controller, microcontroller, central processing unit, digital signal processor, application specific integrated circuit, field programmable gate array, other electronic circuitry, a combination of one or more of the preceding) coupled to one or more machine-readable storage media to store code for execution on the set of processors and / or to store data. For instance, an electronic device may include non-volatile memory containing the code since the non-volatile memory can persist code / data even when the electronic device is turned off (when power is removed), and while the electronic device is turned on that part of the code that is to be executed by the processor(s) of that electronic device is typically copied from the slower nonvolatile memory into volatile memory (e g., dynamic random access memory (DRAM), static random access memory (SRAM)) of that electronic device. Typical electronic devices also include a set of one or more physical network interface(s) (NI(s)) to establish network connections (to transmit and / or receive code and / or data using propagating signals) with other electronic devices. For example, the set of physical NIs (or the set of physical NI(s) in combination with the set of processors executing code) may perform any formatting, coding, or translating to allow the electronic device to send and receive data whether over a wired and / or a wireless connection. In some embodiments, a physical NI may comprise radio circuitry capable of receiving data from other electronic devices over a wireless connection and / or sending data out to other devices via a wireless connection. This radio circuitry may include transmitted s),receiver(s), and / or transceiver s) suitable for radiofrequency communication. The radio circuitry may convert digital data into a radio signal having the appropriate parameters (e.g., frequency, timing, channel, bandwidth, etc.). The radio signal may then be transmitted via antennas to the appropriate recipient(s). In some embodiments, the set of physical NI(s) may comprise network interface controller(s) (NICs), also known as a network interface card, network adapter, or local area network (LAN) adapter. The NIC(s) may facilitate in connecting the electronic device to other electronic devices allowing them to communicate via wire through plugging in a cable to a physical port connected to a NIC. One or more parts of an embodiment of the invention may be implemented using different combinations of software, firmware, and / or hardware.

[0039] A network device (ND) is an electronic device that communicatively interconnects other electronic devices on the network (e.g., other network devices, end-user devices). Some network devices are “multiple services network devices” that provide support for multiple networking functions (e.g., routing, bridging, switching, Layer 2 aggregation, session border control, Quality of Service, and / or subscriber management), and / or provide support for multiple application services (e.g., data, voice, and video).

[0040] As used herein, a pod may refer to smallest deployable unit of computing that may be created and managed in a container orchestration system (COS), such as Kubernetes, Openshift, Nomad (by Hasicorp), Docker Swarm, etc. A pod may include one or more virtual containers with shared storage and network resources, and a specification for how to run the virtual container(s). A pod’s content may be co-located, co-scheduled and run in a shared context.

[0041] As mentioned above, existing approaches for generating a sector carrier (SC) reassignment plan model the problem as a linear programming problem and solve the linear programming problem using exact optimization solvers. While such approaches can provide a globally optimal solution, they do not scale well (e.g., when there are hundreds or thousands of sector carriers).

[0042] Embodiments are disclosed herein that can generate a sector carrier reassignment plan. An embodiment is a method performed by a computing device to generate a sector carrier reassignment plan in a telecommunication network. The method includes obtaining a distance matrix of a plurality or SCs, generating a plurality of fixed clusters of sector carriers using an agglomerative clustering algorithm, the distance matrix, and a current sector carrier assignment, generating, for each of the plurality of fixed clusters, a list of candidate sector carriers of the fixed cluster using the agglomerative clustering algorithm, adding one or more candidate sector carriers included in the lists of candidate sector carriers to corresponding fixed clusters, adding any sector carriers that have not yet been added to a fixed cluster to one of the plurality of fixed clusters, and generating the sector carrier reassignment plan based on the plurality of fixedclusters. Embodiments generate a sector carrier reassignment plan in manner tries to minimize the number of sector carrier migrations, does not exceed the capacity limit of the servers, and tries to satisfy the pairwise affinity and / or anti-affinity preferences between SCs. Also, by using an iterative clustering approach, embodiments scale better compared to existing approaches that model the problem as a linear programming problem.

[0043] Embodiments model / encode the pairwise affinity and / or anti-affinity preferences between SCs using a distance matrix and apply an agglomerative clustering algorithm to the distance matrix in an iterative manner to control the number of SCs in each cluster since each cluster has limited capacity in terms of the number of SCs it can include (the clusters may represent servers and each server has limited capacity). At each iteration, a cluster may be fixed by performing similarity matching with the current SC assignment clusters and selecting SCs that exist in the matched cluster A list of candidate SCs may be generated for each fixed cluster by applying the agglomerative clustering algorithm to a distance matrix of the SCs included in the fixed cluster and any SCs that have not yet been added or “fixed” to a fixed cluster). The SCs that are grouped together with the fixed cluster’s SCs are designated as candidates of that fixed cluster. Candidate SCs may be added to corresponding fixed clusters based on certain rules that take the current SC assignment and the initial clustering result generated by the agglomerative clustering algorithm into consideration. A SC reassignment plan may be generated based on the resulting fixed clusters. Embodiments generate a SC reassignment plan in a manner that tries to minimizes the number of migrations and tries to satisfy the pairwise affinity and anti-affinity requirements / preferences between SCs, while ensuring that the servers do not exceed their maximum capacity.

[0044] Embodiments take into consideration several constraints such as resiliency, mobility, server utilization, server capacity, and number of migrations. Resiliency and mobility may be considered as soft constraints, meaning that they are satisfied as much as possible. Resiliency and mobility preferences may be modeled using a distance matrix. The resiliency preference indicates that certain SCs should be placed apart. The mobility preference indicates that certain SCs (e.g., those with high / medium mobility requirements) should be placed together (e.g., on the same server or pod). Various affinity and anti-affinity preferences between SCs can be flexibly configured / encoded in the distance matrix. Server utilization may be modeled by estimating the minimum number of clusters that are needed to accommodate the SCs. The server capacity may be modeled by applying the agglomerative clustering algorithm in an iterative manner, and at each iteration, fixing a cluster such that it has a size that is less than or equal to the maximum capacity of the cluster (where the maximum capacity of the cluster corresponds to the maximum capacity of a server). The number of migrations may be minimized by trying to assign an SC toa fixed cluster that is most similar to the cluster that includes the SC in the current SC assignment.

[0045] Embodiments generate a sector carrier reassignment plan that minimizes the number of servers needed to host the SCs. As such, embodiments can generate a sector carrier reassignment plan without knowing the exact number of servers available in the infrastructure. Also, embodiments may help reduce the number of servers needed to host SCs

[0046] By iteratively applying a clustering algorithm, embodiments can reduce the execution time and scale better. The use of iterative clustering allows for solving the problem for large number of sector carriers, servers, and pods. The use of an iterative approach allows for taking resource capacity limits or connection limits or a flexible combination of both at the server and pod level.

[0047] Comparing newly generated clusters with current SC assignment (before migration) allows reducing number of migration, hence, cost associated with migration.

[0048] Embodiments are further described herein with reference to the accompanying figures.

[0049] Figure 1 is a diagram showing a system that can generate a sector carrier reassignment plan, according to some embodiments.

[0050] As shown in the diagram, the system includes a SC reassignment manager 110 and servers 120A-E. A server 120 may be used to host one or more SCs. For example, in the example shown in the diagram, server 120A hosts SC-19, server 120B hosts SC-10, SC14, and SC-15, server 120C hosts SC-8 and SC-16, server 120D hosts SC-5 and SC-7, and server 120E hosts SC-9 and SC- 18.

[0051] As will be described in additional detail herein, the SC reassignment manager 110 may generate a SC reassignment plan 185. The SC reassignment manager 110 may receive as input information regarding the current SC assignment 130, information regarding the physical system 135, and information regarding the SC requirements 140. The information regarding the current SC assignment 130 may include information regarding the current assignment of SCs to servers 120. For example, the information regarding the current SC assignment 130 may indicate that server 120A hosts SC-19, server 120B hosts SC-10, SC-14, and SC-15, server 120C hosts SC-8 and SC-16, server 120D hosts SC-5 and SC-7, and server 120E hosts SC-9 and SC- 18. The information regarding the physical system 135 may include information regarding the number of servers 120 available in the physical system and / or information regarding the maximum capacities of the servers 120 available in the physical system (e.g., the maximum number of SCs that each server can host). The information regarding the SC requirements may include information regarding the pairwise affinity and antiaffinity preferences between of SCs (e.g., mobility preferences and / or resiliency preferences).

[0052] In an embodiment, the SC reassignment manager 110 may generate a SC reassignment plan 185 using four phases. In phase one, the SC reassignment manager 110 generates a distance matrix (operation 150). In phase two, the SC reassignment manager 110 generates fixed clusters (operation 160). In phase three, the SC reassignment manager 110 generates a list of candidate sector carriers of fixed clusters (operation 170). In phase four, the SC reassignment manager 110 adds sector carriers (including the candidate sector carriers determined in phase three) to fixed clusters (operation 178). The SC reassignment manager 110 may then generate a SC reassignment plan 185 based on the resulting fixed clusters. The SC reassignment plan 185 may be provided to a deployment unit 195. The deployment unit 195 may effectuate the SC reassignment plan 185 by causing SCs to be reassigned in accordance with the SC reassignment plan 185.

[0053] For example, as shown by the dashed arrows in the diagram, the deployment unit 195 may cause SC- 19 to be reassigned from being hosted by server 120A to being hosted by server 120C, cause SC-9 to be reassigned from being hosted by server 120E to being hosted by server 120C, and cause SC-18 to be reassigned from being hosted by server 120E to being hosted by server 120B. As will be described further herein, the SC reassignment manager 110 may generate the SC reassignment plan 185 in a manner that considers server utilization, the number of migrations of SCs between servers 120, and / or the pairwise affinity and / or antiaffinity preferences between SCs.

[0054] Figure 2 is a flow diagram showing operations of phases one and two of generating a SC reassignment plan, according to some embodiments.

[0055] The operations in the flow diagrams will be described with reference to the exemplary embodiments of the other figures. However, it should be understood that the operations of the flow diagrams can be performed by embodiments of the invention other than those discussed with reference to the other figures, and the embodiments of the invention discussed with reference to these other figures can perform operations different than those discussed with reference to the flow diagrams.

[0056] As mentioned above, in phase one, the SC reassignment manager generates a distance matrix and in phase two, the SC reassignment manager generates fixed clusters. Phase one may include operations 205 and 210. Phase two may include operations 215-270.

[0057] At operation 205, the SC reassignment manager obtains the current SC assignment and the maximum cluster size (m) The maximum cluster size, m, represents the maximum capacity of the servers in terms of the number of SCs that can be hosted on a given server. For sake of simplicity, the example assumes that all of the clusters / servers have the same maximum capacity of m. However, those of ordinary skill in the art will appreciate that the techniques disclosedherein can be modified / adapted to accommodate scenarios where different clusters / servers have different maximum capacities.

[0058] At operation 210, the SC reassignment manager generates a distance matrix of SCs. In an embodiment, the distance matrix of the SCs reflects pairwise affinity preferences and antiaffinity preferences between SCs. An example of an affinity preference is a mobility preference. Mobility preferences prefer that SCs be placed close to each other. Thus, a pair of SCs with a relatively high degree of mobility preference with respect to each other may be assigned a relatively small distance value in the distance matrix. An example of an anti-affinity preference is a resiliency preference. Resiliency preferences prefer that SCs be placed apart from each other. Thus, a pair of SCs with a relatively high degree of resiliency preference with respect to each other may be assigned a relatively larger distance value in the distance matrix. When both types of preferences co-exist, the difference between the two values may be taken and assigned to pair. This way, the mobility preference may work against the resiliency preference. An example distance matrix is shown in Figure 6. An example of how to generate a distance matrix is described in the Patent Cooperation Treaty (PCT) application PCT / IB2023 / 051338 titled, “METHODS AND APPARATUS FOR LARGE-SCALE SECTOR CARRIER ASSIGNMENT,” which is incorporated herein in its entirety by reference.

[0059] At operation 215, the SC reassignment manager determines the minimum number of clusters, n. The value of “n” may represent the number of servers that will host SCs after the SCs are reassigned. In an embodiment, the value of n is determined to be number SCs I m, where number SCs represents the number of SCs that are to be hosted by the servers after the reassignment and m is the maximum cluster size. In an embodiment, the value of n is provided to the SC reassignment manager as an input.

[0060] At operation 220, the SC reassignment manager applies an agglomerative clustering algorithm to the distance matrix to generate n clusters. An agglomerative clustering algorithm is a type of hierarchical clustering algorithm that divides the elements included in a distance matrix into several clusters such that elements in the same cluster are closer in terms of the distance indicated in the distance matrix. Various agglomerative clustering algorithms are known in the art, and thus are not described in detail herein in the interest of brevity. The output of operation 220 may be referred to as the “initial clustering result” and may be used in phase four, as will be further described herein.

[0061] At operation 230, the SC reassignment manager determines whether there exists a cluster in the clustering result that has a cluster size that is larger than m (cluster size > m, where cluster size is the size of the cluster in terms of the number of elements / SCs included in the cluster and m is the maximum cluster size).

[0062] If there exists a cluster that has a cluster size that is larger than m, then at operation 235, the SC reassignment manager selects a cluster, x, having a cluster size that is larger than m. Otherwise, if there is no cluster that has a cluster size that is larger than m, then at operation 240, the SC reassignment manager may select the largest cluster (so cluster x is the largest cluster in the clustering result).

[0063] At operation 243, the SC reassignment manager determines which cluster in the current SC assignment is most similar to cluster x in terms of SC membership (e.g., which cluster in the current SC assignment includes the most number of SCs that overlap with the SCs included in cluster x).

[0064] At operation 245, the SC reassignment manager fixes the SCs of cluster x such that its cluster size is less than or equal to m (cluster size <= m). The SC reassignment manager may fix the SCs in cluster x that are also included in the cluster in the current SC assignment that is most similar to cluster x (fix the SCs that overlap between cluster x and the cluster in the current SC assignment that is most similar to cluster x).

[0065] At operation 250, the SC reassignment manager updates the distance matrix to exclude the fixed SCs.

[0066] At operation 255, the SC reassignment manager decrements n by one if ft is greater than two (n = n - 1 if n > 2).

[0067] At operation 260, the SC reassignment manager determines whether n has been equal to two (n = 2) for two consecutive iterations. If n has not been equal to two for two consecutive iterations, then the SC reassignment manager repeats operations 220-255 (using the updated / decremented value of ri). Otherwise, if n has been equal to two for two consecutive iterations, then at operation 265, the SC reassignment manager groups the remaining SCs (that were not fixed to a fixed cluster) into a list of remaining SCs. If n has been equal to two for two consecutive iterations, this means u fixed clusters have been generated so no new fixed clusters need to be generated. At operation 270, the SC reassignment manager returns the fixed clusters and the list of remaining SCs.

[0068] It should be noted that in this example u is not allowed to be less than 2 because grouping all SCs into a single cluster may cause some of the pairwise anti-affinity preferences between SCs (e.g., resiliency preferences) encoded in the distance matrix to be lost.

[0069] Figure 3 is a flow diagram showing operations of phase three of generating a SC reassignment plan, according to some embodiments.

[0070] As mentioned above, in phase three, the SC reassignment manager generates a list of candidate sector carriers for each fixed cluster.

[0071] At operation 305, the SC reassignment manager obtains the fixed clusters and the list of remaining SCs that were generated in phase two.

[0072] At operation 310, the SC reassignment manager obtains the next fixed cluster, y, that has not been processed in phase three yet.

[0073] At operation 315, the SC reassignment manager generates a distance matrix of the SCs included in fixed cluster; / and the SCs included in the list of remaining SCs.

[0074] At operation 320, the SC reassignment manager applies an agglomerative clustering algorithm to the distance matrix (that was generated in operation 315) to generate two clusters.

[0075] At operation 325, the SC reassignment manager determines the SCs in the two clusters that are grouped together with the SCs included in fixed cluster

[0076] At operation 330, the SC reassignment manager designates the SCs grouped together with the SCs included in fixed clustery as candidates of fixed clustery

[0077] At operation 335, the SC reassignment manager determines whether there are more fixed clusters to process in phase three. If there are more fixed clusters to process in phase three, the SC reassignment manager performs operations 310-330 for the next fixed cluster. Otherwise, if there are no more fixed clusters to process, phase three ends.

[0078] Figure 4 is a flow diagram showing operations of phase four of generating a SC reassignment plan, according to some embodiments.

[0079] As mentioned above, in phase four, the SC reassignment manager adds sector carriers to fixed clusters based on rules. While certain rules are described herein to illustrate a particular embodiment, it should be appreciated that other embodiments may use other rules depending on the requirements / goals of the system.

[0080] At operation 405, the SC reassignment manager obtains the fixed clusters, the list of candidates of fixed clusters, the current SC assignment, the initial clustering result, and the list of remaining SCs.

[0081] The fixed clusters and the list of remaining SCs may have been generated in phase two. The current SC assignment may be one of the inputs provided to the SC reassignment manager. The initial clustering result may be the result of the initial clustering performed in phase two. The list of candidates of fixed clusters may have been generated in phase three.

[0082] At operation 410, the SC reassignment manager identifies unique candidates of fixed clusters. A unique candidate is a SC that is a candidate for only one fixed cluster.

[0083] At operation 415, the SC reassignment manager determines whether any of the fixed clusters have a unique candidate and remaining capacity. If none of the fixed clusters have a unique candidate and remaining capacity, then the flow moves to operation 425. Otherwise, if a fixed cluster has a unique candidate and remaining capacity, then at operation 420, the SCreassignment manager adds the unique candidate to the fixed cluster. The SC reassignment manager also removes that unique candidate from the list of remaining SCs.

[0084] At operation 425, the SC reassignment manager identifies single candidates of fixed clusters. A single candidate is a SC that is the only a candidate for a fixed cluster. At operation 430, the SC reassignment manager determines whether any of the fixed clusters have a single candidate and remaining capacity. If none of the fixed clusters have a single candidate and remaining capacity, then the flow moves to operation 440. Otherwise, if a fixed cluster has a single candidate and remaining capacity, then at operation 435, the SC reassignment manager adds the single candidate to the fixed cluster. The SC reassignment manager also removes that single candidate from the list of remaining SCs. In an embodiment, if a SC is a single candidate for more than one fixed cluster, then it can be added to any one of those fixed clusters (e.g., the cluster can be randomly selected).

[0085] At operation 440, the SC reassignment manager selects the next SC in the list of remaining SCs. The SC reassignment manager performs operations 445-455 for each SC in the list of remaining SCs.

[0086] At operation 445, the SC reassignment manager determines the cluster, c, in the current SC assignment that includes the SC. At operation 450, the SC reassignment manager determines the fixed cluster that is most similar to cluster c (in terms of SC membership). At operation 455, the SC reassignment manager determines whether the most similar fixed cluster has remaining capacity. If the most similar fixed cluster has remaining capacity, then at operation 460, the SC reassignment manager adds the SC to the most similar fixed cluster. The SC reassignment manager also removes the SC from the list of remaining SCs. Otherwise, if the most similar fixed cluster does not have remaining capacity, then the flow moves back to operation 440 to process the next SC (if there are any unprocessed SCs in the remaining list of SCs).

[0087] Operations 440-460 help minimize the number of SC migrations by adding SCs to fixed clusters that are most similar to the cluster that includes the SCs in the current SC assignment.

[0088] At operation 465, selects the next SC in the list of remaining SCs. The SC reassignment manager performs operations 470-480 for each SC in the list of remaining SCs.

[0089] At operation 470, the SC reassignment manager determines the cluster c in the initial clustering result (e.g., that was generated in phase two) that includes the SC. At operation 475, the SC reassignment manager determines the fixed cluster that is most similar to cluster c. At operation 480, the SC reassignment manager determines whether the most similar fixed cluster has remaining capacity. If the most similar fixed cluster has remaining capacity, then atoperation 485, the SC reassignment manager adds the SC to the most similar fixed cluster. The SC reassignment manager also removes the SC from the list of remaining SCs. Otherwise, if the most similar fixed cluster does not have remaining capacity, then the flow moves back to operation 465 to process the next SC (if there are any unprocessed SCs in the remaining list of SCs).

[0090] Operations 465-485 help with satisfying the pairwise affinity and / or anti-affinity preferences between SCs encoded in the distance matrix (e.g., mobility and resiliency preferences) by adding SCs to fixed clusters that are most similar to the cluster that included the SC in the initial clustering result.

[0091] At operation 490, the SC reassignment manager adds any remaining SCs in the list of remaining SCs to fixed clusters with remaining capacity (e.g., randomly or starting with the clusters having the most capacity remaining).

[0092] Figures 5-17 show an example of how a SC reassignment plan can be generated.

[0093] Figure 5 is a diagram showing a current SC assignment and a SC reassignment, according to some embodiments.

[0094] The example assumes that there are 20 SCs (SCs that are numbered 1-20) but half of them are removed from the system for some reason. The SCs that are removed are the SCs numbered 4, 13, 17, 12, 1, 0, 11, 3, 2, and 6. As a result, the SCs in the current SC assignment 520 include the SCs numbered 19, 10, 14, 15, 8, 16, 5, 7, 9, and 18.

[0095] As shown in the diagram, in the current SC assignment 520, server-1 hosts SC 19, server-2 hosts SCs 10, 14, and 15, server-3 hosts SCs 8 and 16, server-4 hosts SCs 5 and 7, and server-5 hosts SCs 9 and 18. The maximum capacity of a server in this example is four SCs. That is, each server can host up to a maximum of four SCs. Each server may correspond to a cluster (and thus the maximum capacity of a cluster is also four SCs).

[0096] Embodiments disclosed herein can be used to reassign the SCs, as shown by the SC reassignment 530. After the SCs are reassigned, server-2 hosts SCs 10, 14, 15, and 18, server-3 hosts SCs 8, 9, 16, and 19, and server-4 hosts SCs 5 and 7. It is noted that after the SCs are reassigned, the SCs are hosted by only three servers (instead of five servers). Thus, the reassignment was able to reduce the number of servers needed to host the SCs while still taking into account the pairwise preferences between the SCs. Also, the reassignment can be reached with only three migrations.

[0097] A more detailed description of how embodiments are able to determine the SC reassignment 530 is provided below.

[0098] Figure 6 is a diagram showing a distance matrix, according to some embodiments.

[0099] As mentioned above, a distance matrix is generated in phase one. The diagram shows a distance matrix 610 before SCs are removed and a distance matrix 620 after the ten SCs (SCs 4, 13, 17, 12, 1, 0, 11, 3, 2, 6) are removed. Each element in the distance matrix indicates the distance between corresponding SCs. For example, distance matrix 620 indicates that the distance between SC 5 and itself is 0, the distance between SC 5 and SC 7 is 1, the distance between SC 5 and SC 8 is 5, and so on The distance matrix may encode pairwise affinity and anti-affinity preferences between pairs of SCs, as described above.

[0100] Figure 7 is a diagram showing the generation of the first fixed cluster, according to some embodiments.

[0101] As mentioned above, during phase two, fixed clusters are generated such that they do not exceed the maximum cluster size.

[0102] The minimum number of clusters needed to fit all ten SCs is determined. Assuming the maximum capacity of each cluster is four SCs, the minimum number of clusters needed is three. Thus, n is set to three in this example.

[0103] As shown in the diagram, the list of remaining SCs includes SCs 5, 7, 8, 9, 10, 14, 15, 16, 18, and 19. These SCs are clustered into three clusters (since n = 3) by applying an agglomerative clustering algorithm to the distance matrix 620. In this example, the clustering produces a clustering result that includes a first cluster that includes SCs 15 and 18, a second cluster that includes SCs 8, 10, 14, 16, and 19, and a third cluster that includes SCs 5, 7, and 9. This clustering result may be referred to as the initial clustering result and may be used in phase four.

[0104] The second cluster (which includes SCs 8, 10, 14, 16, 19) exceeds the maximum capacity of the cluster (it includes more than four SCs). It is desirable to reduce the size of this cluster and “fix” some of its SCs. This cluster is selected and compared against the clusters in the current SC assignment 520 to determine the most similar cluster in the current SC assignment 520 (in terms of SC membership). In this example, the cluster in the current SC assignment 520 that includes SCs 10, 14, and 15 and the cluster in the current SC assignment 520 that includes SCs 8 and 16 are the most similar, as forty percent of the SCs included in the cluster exceeding the capacity are also included in the current SC assignment clusters.

[0105] One of these current SC assignment clusters can be selected using a tie breaking mechanism. Various tie breaking mechanisms can be used for this purpose. One possible strategy is to randomly select one of the clusters. Another possible strategy is to select the cluster that is bigger in size (e.g., select the cluster that includes SCs 10, 14, and 15 in thisexample to give a chance for SC 15 to join this cluster later). In this example, the cluster in the current SC assignment that includes SCs 10, 14, and 15 is selected.

[0106] A fixed cluster is generated to include the SCs that overlap between the cluster that exceeds the capacity and the selected current SC assignment cluster. In this example, a fixed cluster is generated that includes SCs 10 and 14. The SCs included in the fixed cluster are now considered fixed. As such, these SC may be removed from the list of remaining SCs. Accordingly, in this example, SCs 10 and 14 are removed from the list of remaining SCs.

[0107] Figure 8 is a diagram showing generation of the second fixed cluster, according to some embodiments.

[0108] The second fixed cluster can be generated using similar operations as those described above but after removing the SCs that are fixed and decrementing n by one. In this example, SCs 10 and 14 are removed from the list of remaining SCs (since they were fixed as part of the first fixed cluster) and n is set to two (n = 2). The list of remaining SCs includes SCs 5, 7, 8, 9, 15, 16, 18, and 19. These SCs are clustered into two clusters (since n = 2) using the agglomerative clustering algorithm. In this example, the clustering produces a clustering result that includes a first cluster that includes SCs 5, 7, 9, 15, and 18 and a second cluster that includes SCs 8, 16, and 19.

[0109] The first cluster (which includes SCs 5, 7, 9, 15, and 18) exceeds the maximum capacity of the cluster (it includes more than four SCs). It is desirable to reduce the size of this cluster and “fix” some of its SCs. This cluster is selected and compared against the clusters in the current SC assignment 520 to determine the most similar cluster in the current SC assignment 520 (in terms of SC membership). In this example, the cluster in the current SC assignment 520 that includes SCs 5 and 7 and the cluster in the current SC assignment 520 that includes SCs 9 and 18 are the most similar, as forty percent of the SCs included in the cluster exceeding the capacity are also included in the current SC assignment clusters.

[0110] In this example, the cluster that includes SCs 5 and 7 is selected (e.g., using a tiebreaking mechanism). A fixed cluster is generated to include the SCs that overlap between the cluster that exceeds the capacity and the selected current SC assignment cluster. In this example, a fixed cluster is generated that includes SCs 5 and 7. The SCs included in the fixed cluster are now considered fixed. As such, these SC may be removed from the list of remaining SCs. Accordingly, in this example, SCs 5 and 7 are removed from the list of remaining SCs.

[0111] Figure 9 is a diagram showing the generation of the third fixed cluster, according to some embodiments.

[0112] The third fixed cluster can be generated using similar operations as those described above but after removing the SCs that are fixed. In this example, SCs 5 and 7 are furtherremoved from the list of remaining SCs (since they were fixed as part of the second fixed cluster) and n is set to two (n = 2). It is noted that n is not decremented to be less than two, as setting n to one could cause some pairwise preferences between SCs (e.g., resiliency preferences) to be lost. The list of remaining SCs includes SCs 8, 9, 15, 16, 18, and 19. These SCs are clustered into two clusters (since n = 2) using the agglomerative clustering algorithm. In this example, the clustering produces a clustering result that includes a first cluster that includes SCs 8, 9, 16, and 19 and a second cluster that includes SCs 15 and 18.

[0113] Both clusters do not exceed the maximum capacity of the cluster, but the first cluster (which includes SCs 8, 9, 16, and 19) is selected in this example because it is the larger of the two clusters. It is desirable to “fix” some of the SCs included in the selected cluster. The selected cluster is compared against the clusters in the current SC assignment 520 to determine the most similar cluster in the current SC assignment 520 (in terms of SC membership). In this example, the cluster in the current SC assignment 520 that includes SCs 8 and 16 is the most similar, as fifty percent of the SCs included in the selected cluster are also included in the current SC assignment cluster. Since the selected cluster does not exceed the maximum capacity, another option is to fix all of the SCs included in this cluster. However, in general, an approach that fixes SCs based on comparing the selected cluster against the current SC assignment will provide more flexibility, as the non-fixed SCs can be considered in future steps.

[0114] A fixed cluster is generated to include the SCs that overlap between the largest cluster and the most similar current SC assignment cluster. In this example, a fixed cluster is generated that includes SCs 8 and 16. The SCs included in the fixed cluster are now considered fixed. As such, these SC may be removed from the list of remaining SCs. Accordingly, in this example, SCs 8 and 16 are removed from the list of remaining SCs.

[0115] Figure 10 is a diagram showing fixed clusters, according to some embodiments.

[0116] As shown in the diagram, the fixed clusters include the first fixed cluster that includes SCs 10 and 14, the second fixed cluster that includes SCs 5 and 7, and the third fixed cluster that includes SCs 8 and 16. As will be described in additional detail below, additional SCs can be added to the fixed clusters based on certain rules.

[0117] Figure 11 is a diagram showing generating candidates of the first fixed cluster, according to some embodiments.

[0118] As mentioned above, in phase three, a list of candidate SCs is generated for each fixed cluster. Candidates SCs may be added to corresponding fixed clusters, as will be described in additional detail herein.

[0119] The diagram shows an example of generating a list of candidate SCs for the first fixed cluster (that includes SCs 10 and 14). At this stage, the list of remaining SCs includes SCs 9,15, 18, and 19. An agglomerative clustering algorithm is applied to a distance matrix of the SCs included in the first fixed cluster and the SCs included in the list of remaining SCs to generate two clusters (n clusters = 2). In this example, the agglomerative clustering is applied to a distance matrix of SCs 10, 14, 9, 15, 18, and 19. The clustering algorithm produces a clustering result that includes a first cluster that includes SCs 9, 10, 14, 15, and 18 and a second cluster that includes SC 19. The first cluster includes the SCs included in the first fixed cluster (SCs 10 and 14). The other SCs that are grouped together with the SCs included in the first fixed cluster are designated as candidates of the first fixed cluster. Accordingly, in this example, SCs 9, 15, and 18 are designated as candidate SCs of the first fixed cluster. Thus, the list of candidate SCs of the first fixed cluster includes SCs 9, 15, and 18.

[0120] The list of candidate SCs of the other fixed clusters can be generated using similar operations as those described above, and are not further described herein in for sake of brevity.

[0121] Figure 12 is a diagram showing the list of candidates of the fixed clusters, according to some embodiments. As shown in the diagram, the list of candidates of the first fixed cluster includes SCs 9, 15, and 18, the list of candidates of the second fixed cluster includes SCs 9 and 19, and the list of candidates of the third fixed cluster includes SCs 9, 15, and 18.

[0122] Figure 13 is a diagram showing the addition of a unique candidate to a fixed cluster, according to some embodiments.

[0123] As mentioned above, in phase four, SCs are added to fixed clusters based on rules. The diagram shows the addition of unique candidates to fixed clusters.

[0124] If a fixed cluster has a unique candidate, then the unique candidate is added to the fixed cluster. In this example, SC 19 is a unique candidate since it is a candidate for only one fixed cluster (the second fixed cluster). Thus, SC 19 is added to the corresponding fixed cluster (the second fixed cluster). A candidate SC is only added to the fixed cluster if the cluster has remaining capacity. In this example, SC 19 is added (and “fixed”) to the second fixed cluster since the second fixed cluster has remaining capacity. As a result, the second fixed cluster now includes three SCs that are “fixed” (SCs 8, 16, and 19). SC 19 is then removed from the list of remaining SCs.

[0125] Figure 14 is a diagram showing adding single candidates, according to some embodiments. The diagram shows the addition of single candidates to fixed clusters as part of phase four.

[0126] If a fixed cluster has a single candidate, then the single candidate is added to the fixed cluster. In this example, SC 9 is a single candidate (since it is the only candidate of the second fixed cluster). Thus, SC 9 is added to the corresponding fixed cluster (the second fixed cluster). A candidate SC is only added to the fixed cluster if the cluster has remaining capacity. In thisexample, SC 9 is added (and “fixed”) to the second fixed cluster because the second fixed cluster has remaining capacity. As a result, the second fixed cluster now includes four SCs that are “fixed” (SCs 8, 9, 16, and 19). SC 9 is then removed from the list of remaining SCs. It is noted that the second fixed cluster has reached its maximum capacity (four SCs) and thus no more SCs can be added to the second fixed cluster.

[0127] Figure 15 is a diagram showing the addition of an SC to a fixed cluster that is most similar to a cluster that includes the SC in the current SC assignment, according to some embodiments.

[0128] Continuing with the example, at this stage, there are still two SCs included in the list of remaining SCs (SCs 15 and 18).

[0129] SC 15 is first selected for processing. The cluster that includes SC 15 in the current SC assignment 520 is identified. In this example, this is the cluster that includes SCs 10, 14, and 15. The fixed cluster that is most similar to the identified cluster is determined. In this example, the first fixed cluster (that includes SCs 10 and 14) is determined to be most similar to the current SC assignment cluster (there is a 66% match). Thus, SC 15 is added to the first fixed cluster. As a result, the first fixed cluster now includes SCs 10, 14, and 15. SC 15 is then removed from the list of remaining SCs.

[0130] Similar operations are performed for SC 18. The SC that includes SC 18 in the current SC assignment 520 is the cluster that includes SCs 9 and 18. The fixed cluster that is most similar to this cluster is the second fixed cluster that includes SCs 8, 9, 16, and 19. However, since the second fixed cluster does not have remaining capacity (it already includes four SCs), SC 18 cannot be added to this fixed cluster.

[0131] By adding an SC to a fixed cluster that is most similar to the cluster that includes the SC in the current SC assignment, the number of SC migrations can be minimized or reduced.

[0132] Figure 16 is a diagram showing the addition of an SC to a fixed cluster that is most similar to a cluster that includes the SC in an initial clustering result, according to some embodiments.

[0133] Continuing with the example, at this stage, there is only one SC included in the list of remaining SCs: SC 18.

[0134] The cluster that includes SC 18 in the initial clustering result is identified. In this example, this is the cluster that includes SCs 15 and 18. The fixed cluster that is most similar to this cluster is determined. In this example, the first fixed cluster (that includes SCs 10, 14, and 15) is determined to be most similar to the initial clustering result cluster (there is a 50% match). The first fixed cluster has remaining capacity. Thus, SC 18 is added to the first fixedcluster. As a result, the first fixed cluster now includes SCs 10, 14, 15, and 18. SC 18 is then removed from the list of remaining SCs.

[0135] Since there are no more SCs remaining in the list of remaining SCs, the process ends. Figure 17 shows the final fixed clusters, which can be used to generate the SC reassignment plan.

[0136] By adding an SC to a fixed cluster that is most similar to a cluster that includes the SC in an initial clustering result, pairwise affinity and / or anti-affinity preferences encoded in the distance matrix (e.g., mobility and resiliency preferences) can be taken into consideration.

[0137] Figure 17 is a diagram showing reassigned clusters, according to some embodiments.

[0138] The reassigned clusters 530 (the SC reassignment) include a first cluster that includes SCs 10, 14, 15, 18, a second cluster that includes SCs 8, 9, 16, and 19, and a third cluster that includes SCs 5 and 7. It is noted that the number of clusters was reduced from five to three, and only three SC migrations are needed to effectuate the reassignment. A SC reassignment plan can be generated based on the fixed clusters. For example, a SC reassignment plan can be generated that places SCs 10, 14, 15, and 18 together on the same server, places SCs 8, 9, 16, and 19 together on the same server, and SCs 5 and 7 together on the same server.

[0139] Figure 18 is a diagram showing an example hierarchy of computing resources, according to some embodiments.

[0140] The SC (re)assignment problem can be modeled as a hierarchical bin packing problem, where SCs of the telecommunication network are assigned physical resources and / or virtual resources provided by an electronic device or other device of the telecommunication network. The telecommunication network may have a hierarchy of computing resources such as the hierarchy shown in the diagram.

[0141] As shown in the diagram, the hierarchy includes a hub 1801-1, a plurality of j physical servers 1802-1, 1802-2, ..., 1802-j that are each connected with the hub 1801-1, and a plurality of virtual containers / pods 1803-1, 1803-2, ..., 1803 -k implemented on various ones of the plurality of the physical servers 1802-1, 1802-2, ..., 1802-j.

[0142] In some embodiments (not shown in the diagram), the hierarchy further includes another hub (a second hub) that is arranged at the same level as hub 1801-1, and a second plurality of p servers that are each connected with the second, and a second plurality of pods implemented on various ones of the second plurality of servers. The hierarchy may further include a master hub (not shown in the diagram) that is arranged at a higher level than hub 1801- 1 and that connects to the hub 1801-1 and one or more other hubs. In some cases, the master hub may provide supervisory functionality to the hub 1801-1 and other hubs at the same level as hub 1801-1 (e.g., coordinating and / or controlling operation). The physical servers 1802 mayprovide physical resources and the virtual container / pods 1803 may provide virtualized resources. In an embodiment, one or more of the physical servers 1802 implement virtual machines (VM), wherein the VMs form an intermediate virtualization level between a physical server and one or more pods implemented on the physical server. In some embodiments, all of the pods are implemented directly on the respective servers (i.e., without an intermediate virtualization level).

[0143] Embodiment disclosed herein may be used to reassign SCs to physical servers. The SCs assigned to a server may then be assigned to a container / pod implemented by that server using existing mechanisms such as the second stage of the two-stage optimization approach described in PCT application PCT / IB2022 / 060882 titled, “METHOD AND APPARATUS FOR SECTOR CARRIER ASSIGNMENT,” which is incorporated herein in its entirety by reference.

[0144] Figure 19 is a flow diagram of a method for generating a sector carrier reassignment plan, according to some embodiments. In an embodiment, the method is performed by a computing device (e.g., that implements a SC reassignment manager 110).

[0145] At operation 1910, the computing device obtains a distance matrix of a plurality of sector carriers. In an embodiment, the distance matrix of the plurality of sector carriers reflects pairwise affinity preferences and / or anti-affinity preferences between sector carriers.

[0146] At operation 1920, the computing device generates a plurality of fixed clusters of sector carriers using an agglomerative clustering algorithm, the distance matrix, and a current sector carrier assignment.

[0147] At operation 1930, the computing device generates, for each of the plurality of fixed clusters, a list of candidate sector carriers of the fixed clusters using the agglomerative clustering algorithm, a list of sector carriers included in the fixed cluster, and a list of sector carriers that have not yet been added to a fixed cluster (this list is also referred to herein above as the list of remaining SCs).

[0148] At operation 1940, the computing device adds one or more candidate sector carriers included in the list of candidate sector carriers to corresponding fixed clusters.

[0149] At operation 1950, the computing device adds any sector carriers that have not yet been added to a fixed cluster to one of the plurality of fixed clusters.

[0150] At operation 1960, the computing device generates a sector carrier reassignment plan based on the plurality of fixed clusters.

[0151] Figure 20 is a flow diagram showing operations to generate fixed clusters, according to some embodiments. In an embodiment, the operations shown in the diagram may be performed by a computing device as part of performing operation 1920 shown in Figure 19.

[0152] At operation 2010, the computing device applies the agglomerative clustering algorithm to the distance matrix of the plurality of sector carriers to generate at least two clusters of sector carriers.

[0153] At operation 2020, the computing device selects one of the at least two clusters (e.g., a cluster that has exceeded its capacity or the largest of the at least two clusters).

[0154] At operation 2030, the computing device determines which cluster of sector carriers included in the current sector carrier assignment is most similar to the selected cluster in terms of sector carrier membership.

[0155] At operation 2040, the computing device generates a fixed cluster to include sector carriers that are included in both the selected cluster and the cluster included in the current sector carrier assignment that was determined to be most similar to the selected cluster.

[0156] At operation 2050, the computing device updates the distance matrix of the plurality of sector carriers to exclude the sector carriers included in the fixed cluster.

[0157] The computing device may repeat operations 2010-2050 n-\ more times to generate a total of n fixed clusters.

[0158] Figure 21 is a flow diagram showing operations to generate lists of candidate sector carriers of fixed clusters, according to some embodiments. In an embodiment, the operations shown in the diagram may be performed by a computing device as part of performing operation 1930 shown in Figure 19.

[0159] At operation 2110, the computing device generates a distance matrix of sector carriers included in the fixed cluster and the sector carriers that have not yet been added to a fixed cluster.

[0160] At operation 2120, the computing device applies the agglomerative clustering algorithm to the distance matrix to generate two clusters of sector carriers.

[0161] At operation 2130, the computing device designates any sector carriers in the two clusters of sector carriers that are clustered with the sector carriers included in the fixed cluster as candidate sector carriers of the fixed cluster.

[0162] The computing device may repeat operations 2110-2130 for each fixed cluster to generate a list of candidate sector carriers for each fixed cluster.

[0163] Figure 22 is a flow diagram showing operations to add sector carriers included in the lists of candidate sector carriers to fixed clusters, according to some embodiments. In an embodiment, the operations shown in the diagram may be performed by a computing device as part of performing operation 1940 shown in Figure 19.

[0164] At operation 2210, the computing device determines unique candidate sector carriers.

[0165] At operation 2220, the computing device adds one or more of the unique candidate sector carriers to corresponding fixed clusters responsive to determining that the corresponding fixed cluster has remaining capacity.

[0166] At operation 2230, the computing device removes the added one or more unique candidate sector carriers from the list of sector carriers that not yet been added to a fixed cluster.

[0167] Operations 2220 and 2230 may be skipped if there are no unique candidate sector carriers.

[0168] At operation 2240, the computing device determines single candidate sector carriers.

[0169] At operation 2250, the computing device adds one or more of the single candidate sector carriers to corresponding fixed clusters responsive to determining that the corresponding fixed cluster has remaining capacity.

[0170] At operation 2260, the computing device removes the added one or more single candidate sector carriers from the list of sector carriers that have not yet been added to a fixed cluster.

[0171] Operations 2250 and 2260 may be skipped if there are no single candidate sector carriers.

[0172] Figure 23 is a flow diagram showing operations to add sector carriers that have not yet been added to a fixed cluster to a fixed cluster, according to some embodiments. In an embodiment, the operations shown in the diagram may be performed by a computing device as part of performing operation 1950 shown in Figure 19.

[0173] At operation 2305, the computing device selects a sector carrier from the list of sector carriers that have not yet been added to a fixed cluster.

[0174] At operation 2310, the computing device determines a cluster of sector carriers included in the current sector carrier assignment that includes the selected sector carrier.

[0175] At operation 2315, the computing device determines which of the plurality of fixed clusters is most similar to the determined cluster in terms of sector carrier membership.

[0176] At operation 2320, the computing device adds the selected sector carrier to the fixed cluster that is most similar to the determined cluster responsive to determining that the fixed cluster has remaining capacity.

[0177] At operation 2325, the computing device removes the selected sector carrier from the list of sector carriers that have not yet been added to a fixed cluster.

[0178] The computing device may repeat operations 2305-2325 for each sector carrier remaining in the list of sector carriers that have not yet been added to a fixed cluster. It is noted that some sector carriers might not be added to a fixed cluster due to the most similar fixedcluster not having remaining capacity. These sector carriers may be processed during subsequent operations, as described herein below.

[0179] At operation 2330, the computing device selects a sector carrier from the list of sector carriers that have not yet been added to a fixed cluster.

[0180] At operation 2335, the computing device determines a cluster of sector carriers included in an initial clustering result that includes the selected sector carrier

[0181] At operation 2340, the computing device determines which of plurality of fixed clusters is most similar to the determined cluster in terms of sector carrier membership.

[0182] At operation 2345, the computing device adds the selected carrier to the fixed cluster that is most similar to the determined cluster responsive to determining that the fixed cluster has remaining capacity.

[0183] At operation 2350, the computing device removes the selected sector carrier from the list of sector carriers that have not yet been added to a fixed cluster.

[0184] The computing device may repeat operations 2330-2350 for each sector carrier remaining in the list of sector carriers that have not yet been added to a fixed cluster. It is noted that some sector carriers might not be added to a fixed cluster due to the most similar fixed cluster not having remaining capacity. These sector carriers may be processed during operation 2355, as described herein below.

[0185] At operation 2355, the computing device adds any sector carriers included in the list of sector carriers that have not yet been added to a fixed cluster to one of the plurality of fixed clusters that has remaining capacity.

[0186] Figure 24 illustrates connectivity between network devices (NDs) within an exemplary network, as well as three exemplary implementations of the NDs, according to some embodiments of the invention. Figure 24 shows NDs 2400A-H, and their connectivity by way of lines between 2400A-2400B, 2400B-2400C, 2400C-2400D, 2400D-2400E, 2400E-2400F, 2400F-2400G, and 2400A-2400G, as well as between 2400H and each of 2400A, 2400C, 2400D, and 2400G. These NDs are physical devices, and the connectivity between these NDs can be wireless or wired (often referred to as a link). An additional line extending from NDs 2400 A, 2400E, and 2400F illustrates that these NDs act as ingress and egress points for the network (and thus, these NDs are sometimes referred to as edge NDs; while the other NDs may be called core NDs).

[0187] Two of the exemplary ND implementations in Figure 24 are: 1) a special-purpose network device 2402 that uses custom application-specific integrated-circuits (ASICs) and a special-purpose operating system (OS); and 2) a general purpose network device 2404 that uses common off-the-shelf (COTS) processors and a standard OS.

[0188] The special-purpose network device 2402 includes networking hardware 2410 comprising a set of one or more processor(s) 2412, forwarding resource(s) 2414 (which typically include one or more ASICs and / or network processors), and physical network interfaces (NIs) 2416 (through which network connections are made, such as those shown by the connectivity between NDs 2400A-H), as well as non-transitory machine readable storage media 2418 having stored therein networking software 2420. During operation, the networking software 2420 may be executed by the networking hardware 2410 to instantiate a set of one or more networking software instance(s) 2422. Each of the networking software instance(s) 2422, and that part of the networking hardware 2410 that executes that network software instance (be it hardware dedicated to that networking software instance and / or time slices of hardware temporally shared by that networking software instance with others of the networking software instance(s) 2422), form a separate virtual network element 2430A-R. Each of the virtual network element(s) (VNEs) 2430A-R includes a control communication and configuration module 2432A-R (sometimes referred to as a local control module or control communication module) and forwarding table(s) 2434A-R, such that a given virtual network element(e.g., 2430A) includes the control communication and configuration module (e.g., 2432A), a set of one or more forwarding table(s) (e.g., 2434A), and that portion of the networking hardware 2410 that executes the virtual network element (e.g., 2430A).

[0189] The special-purpose network device 2402 is often physically and / or logically considered to include: 1) a ND control plane 2424 (sometimes referred to as a control plane) comprising the processor(s) 2412 that execute the control communication and configuration module(s) 2432A-R; and 2) a ND forwarding plane 2426 (sometimes referred to as a forwarding plane, a data plane, or a media plane) comprising the forwarding resource(s) 2414 that utilize the forwarding table(s) 2434A-R and the physical NIs 2416. By way of example, where the ND is a router (or is implementing routing functionality), the ND control plane 2424 (the processor(s) 2412 executing the control communication and configuration module(s) 2432A-R) is typically responsible for participating in controlling how data (e.g., packets) is to be routed (e.g., the next hop for the data and the outgoing physical NI for that data) and storing that routing information in the forwarding table(s) 2434A-R, and the ND forwarding plane 2426 is responsible for receiving that data on the physical NIs 2416 and forwarding that data out the appropriate ones of the physical NIs 2416 based on the forwarding table(s) 2434A-R.

[0190] In an embodiment, software 2420 includes code such as SC reassignment component 2423, which when executed by networking hardware 2410, causes the specialpurpose network device 2402 to perform operations of one or more embodiments disclosed herein (e.g., to generate a SC reassignment plan).

[0191] The general purpose network device 2404 includes hardware 2440 comprising a set of one or more processor(s) 2442 (which are often COTS processors) and physical NIs 2446, as well as non-transitory machine readable storage media 2448 having stored therein software 2450. During operation, the processor(s) 2442 execute the software 2450 to instantiate one or more sets of one or more applications 2464A-R. While one embodiment does not implement virtualization, alternative embodiments may use different forms of virtualization. For example, in one such alternative embodiment the virtualization layer 2454 represents the kernel of an operating system (or a shim executing on a base operating system) that allows for the creation of multiple instances 2462A-R called software containers that may each be used to execute one (or more) of the sets of applications 2464A-R; where the multiple software containers (also called virtualization engines, virtual private servers, or jails) are user spaces (typically a virtual memory space) that are separate from each other and separate from the kernel space in which the operating system is run; and where the set of applications running in a given user space, unless explicitly allowed, cannot access the memory of the other processes. In another such alternative embodiment the virtualization layer 2454 represents a hypervisor (sometimes referred to as a virtual machine monitor (VMM)) or a hypervisor executing on top of a host operating system, and each of the sets of applications 2464A-R is run on top of a guest operating system within an instance 2462A-R called a virtual machine (which may in some cases be considered a tightly isolated form of software container) that is run on top of the hypervisor - the guest operating system and application may not know they are running on a virtual machine as opposed to running on a “bare metal” host electronic device, or through paravirtualization the operating system and / or application may be aware of the presence of virtualization for optimization purposes. In yet other alternative embodiments, one, some or all of the applications are implemented as unikemel(s), which can be generated by compiling directly with an application only a limited set of libraries (e.g., from a library operating system (LibOS) including drivers / libraries of OS sendees) that provide the particular OS sendees needed by the application. As a unikemel can be implemented to run directly on hardware 2440, directly on a hypervisor (in which case the unikernel is sometimes described as running within a LibOS virtual machine), or in a software container, embodiments can be implemented fully with unikernels running directly on a hypervisor represented by virtualization layer 2454, unikernels running within software containers represented by instances 2462A-R, or as a combination of unikernels and the above-described techniques (e.g., unikemels and virtual machines both run directly on a hypervisor, unikernels and sets of applications that are run in different software containers).

[0192] The instantiation of the one or more sets of one or more applications 2464A-R, as well as virtualization if implemented, are collectively referred to as software instance(s) 2452. Each set of applications 2464A-R, corresponding virtualization construct (e.g., instance 2462A- R) if implemented, and that part of the hardware 2440 that executes them (be it hardware dedicated to that execution and / or time slices of hardware temporally shared), forms a separate virtual network element(s) 2460A-R.

[0193] The virtual network element(s) 2460A-R perform similar functionality to the virtual network element(s) 2430A-R - e.g., similar to the control communication and configuration module(s) 2432A and forwarding table(s) 2434A (this virtualization of the hardware 2440 is sometimes referred to as network function virtualization (NFV)). Thus, NFV may be used to consolidate many network equipment types onto industry standard high volume server hardware, physical switches, and physical storage, which could be located in Data centers, NDs, and customer premise equipment (CPE). While embodiments of the invention are illustrated with each instance 2462A-R corresponding to one VNE 2460A-R, alternative embodiments may implement this correspondence at a finer level granularity (e.g., line card virtual machines virtualize line cards, control card virtual machine virtualize control cards, etc.); it should be understood that the techniques described herein with reference to a correspondence of instances 2462A-R to VNEs also apply to embodiments where such a finer level of granularity and / or unikernels are used.

[0194] In certain embodiments, the virtualization layer 2454 includes a virtual switch that provides similar forwarding services as a physical Ethernet switch. Specifically, this virtual switch forwards traffic between instances 2462A-R and the physical NI(s) 2446, as well as optionally between the instances 2462 A-R; in addition, this virtual switch may enforce network isolation between the VNEs 2460A-R that by policy are not permitted to communicate with each other (e g., by honoring virtual local area networks (VLANs)).

[0195] In an embodiment, software 2450 includes code such as SC reassignment component 2453, which when executed by hardware 2440, causes the general purpose network device 2404 to perform operations of one or more embodiments disclosed herein (e.g., to generate a SC reassignment plan).

[0196] The third exemplary ND implementation in Figure 24 is a hybrid network device 2406, which includes both custom ASICs / special-purpose OS and COTS processors / standard OS in a single ND or a single card within an ND. In certain embodiments of such a hybrid network device, a platform VM (i.e., a VM that that implements the functionality of the special-purpose network device 2402) could provide for para-virtualization to the networking hardware present in the hybrid network device 2406.

[0197] A network interface (NI) may be physical or virtual; and in the context of IP, an interface address is an IP address assigned to a NI, be it a physical NI or virtual NI. A virtual NI may be associated with a physical NI, with another virtual interface, or stand on its own (e.g., a loopback interface, a point-to-point protocol interface). A NI (physical or virtual) may be numbered (a NI with an IP address) or unnumbered (a NI without an IP address). A loopback interface (and its loopback address) is a specific type of virtual NI (and IP address) of a NE / VNE (physical or virtual) often used for management purposes; where such an IP address is referred to as the nodal loopback address. The IP address(es) assigned to the NI(s) of a ND are referred to as IP addresses of that ND; at a more granular level, the IP address(es) assigned to NI(s) assigned to a NE / VNE implemented on a ND can be referred to as IP addresses of that NE / VNE.

[0198] Some portions of the preceding detailed descriptions have been presented in terms of algorithms and symbolic representations of transactions on data bits within a computer memory. These algorithmic descriptions and representations are the ways used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consi stent sequence of transactions leading to a desired result. The transactions are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0199] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the above discussion, it is appreciated that throughout the description, discussions utilizing terms such as "processing" or "computing" or "calculating" or "determining" or "displaying" or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

[0200] The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method transactions. The required structure for avariety of these systems will appear from the description above. In addition, embodiments are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of embodiments as described herein.

[0201] An embodiment may be an article of manufacture in which a non-transitory machine- readable storage medium (such as microelectronic memory) has stored thereon instructions (e g., computer code) which program one or more data processing components (generically referred to here as a “processor”) to perform the operations described above. In other embodiments, some of these operations might be performed by specific hardware components that contain hardwired logic (e.g., dedicated digital filter blocks and state machines). Those operations might alternatively be performed by any combination of programmed data processing components and fixed hardwired circuit components.

[0202] Throughout the description, embodiments have been presented through flow diagrams. It will be appreciated that the order of transactions and transactions described in these flow diagrams are only intended for illustrative purposes and not intended to be limiting. One having ordinary skill in the art would recognize that variations can be made to the flow diagrams.

[0203] In the foregoing specification, embodiments have been described with reference to specific exemplary embodiments thereof. It will be evident that various modifications may be made thereto without departing from the broader spirit and scope of the disclosure provided herein. The specification and drawings are, accordingly, to be regarded in an illustrative sense rather than a restrictive sense.

Claims

CLAIMSWhat is claimed is:

1. A method performed by a computing device to generate a sector carrier reassignment plan in a telecommunication network, the method comprising: obtaining (1910) a distance matrix of a plurality of sector carriers; generating (1920) a plurality of fixed clusters of sector carriers using an agglomerative clustering algorithm, the distance matrix, and a current sector carrier assignment; generating (1930), for each of the plurality of fixed clusters, a list of candidate sector carriers of the fixed cluster using the agglomerative clustering algorithm, a list of sector carriers included in the fixed cluster, and a list of sector carriers that have not yet been added to a fixed cluster; adding (1940) one or more candidate sector carriers included in the lists of candidate sector carriers to corresponding fixed clusters; adding (1950) any sector carriers that have not yet been added to a fixed cluster to one of the plurality of fixed clusters; and generating (1960) the sector carrier reassignment plan based on the plurality of fixed clusters.

2. The method of claim 1, wherein the distance matrix of the plurality of sector carriers reflects affinity preferences and anti-affinity preferences between sector carriers.

3. The method of claim 1, wherein the generating the plurality of fixed clusters comprises: applying (2010) the agglomerative clustering algorithm to the distance matrix of the plurality of sector carriers to generate a first set of at least two clusters of sector carriers; selecting (2020) one of the first set of at least two clusters as a first selected cluster; determining (2030) which cluster of sector carriers included in the current sector carrier assignment is most similar to the first selected cluster in terms of sector carrier membership; and generating (2040) a first fixed cluster to include sector carriers that are included in both the first selected cluster and the cluster included in the current sector carrier assignment that was determined to be most similar to the first selected cluster.

4. The method of claim 3, wherein the generating the plurality of fixed clusters further comprises: updating (2050) the distance matrix of the plurality of sector carriers to exclude the sector carriers included in the first fixed cluster; applying the agglomerative clustering algorithm to the updated distance matrix to generate a second set of at least two clusters of sector carriers; selecting one of the second set of at least two clusters as a second selected cluster; determining which cluster of sector carriers included in the current sector carrier assignment is most similar to the second selected cluster in terms of sector carrier membership; and generating a second fixed cluster to include sector carriers that are included in both the second selected cluster and the cluster included in the current sector carrier assignment that was determined to be most similar to the second selected cluster.

5. The method of claim 1, wherein generating, for each of the plurality of fixed clusters, a list of candidate sector carriers of the fixed cluster comprises: generating (2110) a second distance matrix of sector carriers included in a first fixed cluster and the sector carriers that have not yet been added to a fixed cluster; applying (2120) the agglomerative clustering algorithm to the second distance matrix to generate a first set of two clusters of sector carriers; and designating (2130) any sector carriers in the first set of two clusters of sector carriers that are clustered with the sector carriers included in the first fixed cluster as candidate sector carriers of the first fixed cluster.

6. The method of claim 5, wherein generating, for each of the plurality of fixed clusters, a list of candidate sector carriers of the fixed cluster further comprises: generating a third distance matrix of sector carriers included in a second fixed cluster and the sector carriers that have not yet been added to a fixed cluster; applying the agglomerative clustering algorithm to the third distance matrix to generate a second set of two clusters of sector carriers; and designating any sector carriers in the second set of two clusters of sector carriers that are clustered with the sector carriers included in the second fixed cluster as candidate sector carriers of the second fixed cluster.

7. The method of claim 1, wherein the adding candidate sector carriers in the lists of candidate sector carriers to corresponding fixed clusters comprises: determining (2210) unique candidate sector carriers; adding (2220) one or more of the unique candidate sector carriers to corresponding fixed clusters responsive to determining that the corresponding fixed cluster has remaining capacity; and removing (2230) the added one or more unique candidate sector carriers from the list of sector carriers that have not yet been added to a fixed cluster.

8. The method of claim 7, wherein the adding candidate sector carriers in the lists of candidate sector carriers to corresponding fixed clusters further comprises: determining (2240) single candidate sector carriers; adding (2250) one or more of the single candidate sector carriers to corresponding fixed clusters responsive to determining that the corresponding fixed cluster has remaining capacity; and removing (2260) the added one or more single candidate sector carriers from the list of sector carriers that have not yet been added to a fixed cluster.

9. The method of claim 8, wherein the adding any sector carriers that have not yet been added to a fixed cluster to one of the plurality of fixed clusters comprises: selecting (2305) a first sector carrier from the list of sector carriers that have not yet been added to a fixed cluster; determining (2310) a first cluster of sector carriers included in the current sector carrier assignment that includes the first sector carrier; determining (2315) which of the plurality of fixed clusters is most similar to the first cluster in terms of sector carrier membership; adding (2320) the first sector carrier to the fixed cluster that is most similar to the first cluster responsive to determining that the fixed cluster that is most similar to the first cluster has remaining capacity; and removing (2325) the first sector carrier from the list of sector carriers that have not yet been added to a fixed cluster.

10. The method of claim 9, wherein the adding any sector carriers that have not yet been added to a fixed cluster to one of the plurality of fixed clusters further comprises: selecting (2330) a second sector carrier from the list of sector carriers that have not yet been added to a fixed cluster;determining (2335) a second cluster of sector carriers included in an initial clustering result that includes the second sector carrier; determining (2340) which of the plurality of fixed clusters is most similar to the second cluster in terms of sector carrier membership; adding (2345) the second sector carrier to the fixed cluster that is most similar to the second cluster responsive to determining that the fixed cluster that is most similar to the second cluster has remaining capacity; and removing (2350) the second sector carrier from the list of sector carriers that have not yet been added to a fixed cluster11. The method of claim 10, further comprising: adding (2355) any sector carriers included in the list of sector carriers that have not yet been added to a fixed cluster to one of the plurality of fixed clusters that has remaining capacity.

12. A machine-readable medium comprising computer program code which when executed by a computer device carries out the method steps of any of claims 1-11.

13. A computing device (2404) configured to generate a sector carrier reassignment plan, the computing device comprising: one or more processors (2442); and a non-transitory machine-readable storage medium (2448) that stores instructions, which when executed by the one or more processors, causes the computing device to perform the method steps of any one of claims 1-11.

Images