DQN-based user connection control method and apparatus in a hierarchical mobile edge computing system for mobile IoT services

Abstract

The present invention discloses a method and apparatus for controlling a user connection based on a Deep Q-learning Network (DQN) of a Hierarchical Mobile Edge Computing (HMEC) system for mobile IoT services. According to the present invention, a user connection control apparatus is provided comprising: a processor; and a memory connected to the processor, wherein the memory stores program instructions executed by the processor to collect a state including at least one of the throughput of one or more Slave-MEC Servers (S-MECS) and Master-MEC Servers (M-MECS) for all UEs from an mm-HMEC environment, the throughput of an edge UE whose throughput is less than or equal to a preset value, and a handover cost, and to select an action to determine a bias for an optimal direction beam that maximizes a reward from the collected state based on a Deep Q-learning Network (DQN) model.

Description

Technology Field

[0001] The present invention relates to a DQN-based user connection control method and apparatus for a hierarchical mobile edge computing system for mobile IoT services. Background Technology

[0002] In future computing systems, the availability of computing resources is expected to become an increasingly difficult issue due to the widespread growth in demand for mobile traffic.

[0003] To address this problem, a new computing architecture called Hierarchical Mobile Edge Computing (HMEC) network, consisting of a master-MEC server (M-MEC) and a slave-MEC server (S-MEC), is emerging, and interest in deployment between 30 and 300 GHz (mmWave) is growing.

[0004] mmWave-based HMEC (mm-HMEC) systems have many advantages over existing MEC systems.

[0005] 1) Applying mmWave to the fronthaul link increases the freedom of network configuration without installing wired infrastructure.

[0006] 2) The mmWave frequency band has a wide transmission bandwidth that supports high-speed transmission.

[0007] 3) mmWave supports directional transmission (through large-scale antenna installation), has a small transmission range resulting in less interference, and enables efficient resource allocation by spatially reusing the spectrum in a hierarchical structure.

[0008] 4) By using HMEC, user content can be stored in S-MEC and cached closer to the user, thereby minimizing latency.

[0009] While mm-HMEC offers many advantages, it lacks low-complexity beamforming control techniques using mmWave-based large-scale antennas.

[0010] Due to the short propagation distance and directional antenna angle of mmWave, frequent handovers must be handled, so a smooth handover mechanism between MECs is required. Although various studies have been conducted, the following problems exist.

[0011] First, research is needed on orthogonal or non-orthogonal resource scheduling in the wireless fronthaul between M-MECS and S-MECS, and on wireless access between S-MECS and User Equipment (UE).

[0012] Second, although research has been conducted on handover approaches that consider user perception of mobile UEs, dynamic channel load, and handover overhead, the problem of frequent handovers has not yet been resolved.

[0013] Third, densely deployed S-MECS and highly mobile UEs can cause a much larger number of handovers in such networks.

[0014] Finally, to make the most of the new infrastructure, mobile users must actively use S-MEC, which has a relatively low load. The problem to be solved

[0015] To solve the problems of the aforementioned prior art, the present invention proposes a DQN-based user connection control method and apparatus for a hierarchical mobile edge computing system for mobile IoT services that can simultaneously improve system throughput, edge throughput, and handover efficiency in a mm-HMEC system. means of solving the problem

[0016] To achieve the above-mentioned purpose, according to one embodiment of the present invention, a user connection control device based on a Deep Q-learning Network (DQN) of a Hierarchical Mobile Edge Computing (HMEC) system for mobile IoT services is provided, comprising: a processor; and a memory connected to the processor, wherein the memory stores program instructions executed by the processor to select an action for determining a bias for an optimal direction beam that maximizes a reward from the collected state based on a Deep Q-learning Network (DQN) model, wherein the memory collects a state including at least one of the throughput of one or more Slave-MEC Servers (S-MECS) and Master-MEC Servers (M-MECS) for all UEs from an mm-HMEC environment, the throughput of an edge UE whose throughput is less than or equal to a preset value, and a handover cost, and the memory stores program instructions executed by the processor.

[0017] The user connection control device is an M-MECS connected to the one or more S-MECSs, and the program instructions can determine the bias of each of the one or more S-MECSs based on the DQN model and distribute the determined bias to the one or more S-MECSs.

[0018] The above user connection control device is an S-MECS connected to the M-MECS, and each S-MECS receives measurement information from the UE connected to it, transmits the measurement information to the M-MECS, and receives the status from the M-MECS to determine the bias for its optimal direction beam.

[0019] Each of the above one or more S-MECS and M-MECS notifies the UE connected to it of its bias, and if the first UE detects a pilot signal from an adjacent target MECS that is stronger than the previously connected service MECS due to the determined bias, it reports to its service MECS, and the service S-MECS transmits a request to the target MECS so that a new downlink resource can be allocated between the first UE and the target MECS for performing a handover, and the service MECS can transmit the status of the first UE to the target MECS for continuous data communication between the first UE and the target MECS.

[0020] The above DQN model can determine the bias for the optimal direction beam according to the following mathematical formula.

[0021] [Mathematical Formula]

[0022]

[0023] Objective utility function is time It is defined as the weighted sum of the average throughput of all UEs, and , , are the set of all HMECs, the set of S-MECS, and the set of MECS beams, respectively, and is an indicator vector for the link between UE u and MECS B, and is the minimum throughput required for the edge UE, and

[0024] The above program instructions can determine a bias that maximizes the objective utility function.

[0025] The weighted sum of the above average throughput is defined by the following mathematical formula, and

[0026] [Mathematical Formula]

[0027]

[0028] class are the weights for throughput and handover cost, respectively, and is the bias vector of the MECS beams.

[0029] The hourly average throughput for all UEs is defined by the following mathematical formula, and

[0030] [Mathematical Formula]

[0031]

[0032] is the set of UE u

[0033] hour The average handover cost for all UEs can be defined by the following mathematical formula.

[0034] [Mathematical Formula]

[0035]

[0036] Here is UE The handover for is time It is 1 if it occurs, and 0 otherwise.

[0037] The above state may include a UE information vector assigned to one or more S-MECSs, a UE information vector assigned to the M-MECSs, an average throughput vector for edge UEs in all S-MECSs at time t, and an information vector of edge UEs at time t.

[0038] According to another aspect of the present invention, a user connection control system based on a Deep Q-learning Network (DQN) of a Hierarchical Mobile Edge Computing (HMEC) system for mobile IoT services is provided, comprising: a Master-MEC Server (M-MECS) that collects a state including at least one of throughput for all UEs from an mm-HMEC environment, throughput of an edge UE whose throughput is less than or equal to a preset value, and handover cost; and one or more Slave-MEC Servers (S-MECS) that receive measurement information from one or more UEs connected to themselves, assemble it into a state in a DQN model, and transmit it to the M-MECS, wherein the M-MECS is centrally located or each of the one or more S-MECSs is distributed, and the user connection control system selects an action to determine a bias for an optimal direction beam that maximizes a reward from the collected state based on a Deep Q-learning Network (DQN) model.

[0039] According to another aspect of the present invention, a method for controlling a user connection based on a Deep Q-learning Network (DQN) of a Hierarchical Mobile Edge Computing (HMEC) system for mobile IoT services, comprising a device including a processor and memory, is provided, comprising the steps of: collecting a state including at least one of the throughput of one or more Slave-MEC Servers (S-MECS) and Master-MEC Servers (M-MECS) for all UEs from an mm-HMEC environment, the throughput of an edge UE whose throughput is less than or equal to a preset value, and a handover cost; and selecting a behavior for determining a bias for an optimal direction beam that maximizes a reward from the collected state based on a Deep Q-learning Network (DQN) model. Effects of the invention

[0040] According to the present invention, communication overhead and computational complexity can be reduced through a new centralized and distributed user association control scheme that can maximize the utility of the UE related to system throughput, edge throughput, and handover costs in a mm-HMEC system. Brief explanation of the drawing

[0041] Figure 1 is a diagram illustrating the mm-HMEC system architecture composed of M-MECS and several S-MECS. Figure 2 is a diagram illustrating the DQN-based bias control model architecture for a mm-HMEC system. FIG. 3 shows Algorithm 1 for the DQN model learning process according to the present embodiment. FIG. 4 is a diagram illustrating the overall procedure for centralized bias control and handover according to the present embodiment. FIG. 5 is a diagram illustrating a procedure for DQN-based distributed bias control and handover according to the present embodiment. Figure 6 is a diagram illustrating the system utility of an arbitrary destination-based mobility model. Figure 7 is a diagram illustrating the system utility of an arbitrary direction-based mobility model. Figure 8 is a diagram illustrating a Manhattan mobility model-based system utility. Figure 9 is a diagram illustrating the cumulative distribution function of throughput in an arbitrary destination-based mobility model. Figure 10 is a diagram illustrating the cumulative distribution function of throughput in an arbitrary direction-based mobility model. Figure 11 is a diagram illustrating the cumulative distribution function of throughput in a Manhattan mobility model-based system utility. Figure 12 is a diagram showing the number of handovers in an arbitrary destination-based mobility model. Figure 13 is a diagram showing the number of handovers in an arbitrary direction-based mobility model. Figure 14 is a diagram showing the number of handovers in the Manhattan mobility model. Specific details for implementing the invention

[0042] The present invention is capable of various modifications and may have various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the invention to specific embodiments, and it should be understood that the invention includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the invention.

[0043] The terms used herein are merely for describing specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as “comprising” or “having” are intended to indicate the presence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0044] Furthermore, the components of the embodiments described with reference to each drawing are not limited to the respective embodiments and may be implemented to be included in other embodiments within the scope of maintaining the technical spirit of the present invention. It is also obvious that multiple embodiments may be re-implemented as a single embodiment that integrates multiple embodiments, even if a separate description is omitted.

[0045] Furthermore, in the description referring to the attached drawings, identical components are assigned the same or related reference numerals regardless of drawing symbols, and redundant descriptions thereof are omitted. In describing the present invention, if it is determined that a detailed description of related prior art could unnecessarily obscure the essence of the present invention, such detailed description is omitted.

[0046] Figure 1 is a diagram illustrating the mm-HMEC system architecture composed of M-MECS and several S-MECS.

[0047] Referring to FIG. 1, the system according to the present embodiment is based on an architecture in which M-MECS and S-MECS exchange information using a mmWave-based fronthall link, and M / S-MECS and UE share information using a mmWave-based connection link.

[0048] In such a system, S-MECS operates a directional array antenna consisting of a fixed number of beams, each fixed beam having a specific transmit power and antenna angle.

[0049] It is assumed that each UE is connected to an M-MECS or S-MECS.

[0050] Each UE is a service MECS based on downlink signal strength in the following way Select .

[0051]

[0052] Here, is a set of MECS, and is a beam set mounted on the MECS.

[0053] is a reference signal that receives power from the MECS beam, and is the Range Expansion Bias, which is 0 for M-MECS and for low-power S-MECS.

[0054] When a UE is connected to M-MECS and S-MECS, it is referred to as MUE and SUE, respectively. This user connection method is called CRD (Cell Range Expansion) and is a standard user connection method included in 3GPP Rel. 11.

[0055] Since each MECS beam has a unique range extension bias, the range extension bias User connectivity may vary depending on [this].

[0056] 1) When the virtual bias of the service S-MECS becomes strong, the beam of M-MECS becomes relatively weak, some MUEs can be connected to S-MECS, and UEs can perform a handover from M-MECS to S-MECS.

[0057] 2) When the virtual bias of the service S-MECS weakens, the beam of M-MECS becomes relatively stronger, some SUEs can be connected to M-MECS, and UEs can perform a handover from S-MECS to M-MECS.

[0058] 3) SUE is located in the directional beam boundary region of S-MECS to receive multiple side lobe beams, and SUE can perform handover to other beams of S-MECS.

[0060] This embodiment targets multimedia-based mobile unicast or multicast services, such as VR / AR / Metaverse streaming services, V2X application services, or factory automation services of mm-HMEC, which are expected to become highly dominant services in the future.

[0061] To efficiently support these services, in addition to guaranteeing minimum throughput requirements, the highest possible throughput must be provided from the user's perspective, and the service must be delivered with low handover costs from the system's perspective.

[0062] Using existing range extension technology in 5G causes the following problems.

[0063] 1) If the bias is too high, throughput degradation occurs by accommodating more UEs than S-MECS can handle.

[0064] 2) If the bias is too low, the utilization of S-MECS decreases because UEs at the S-MECS boundary are accommodated in M-MECS.

[0065] 3) If the bias is not well adjusted between beams, frequent or unnecessary handovers occur, or handovers are excessively suppressed, causing the UE to lose the opportunity to connect to MECS that provide higher throughput.

[0066] Accordingly, the present embodiment proposes a system in which each beam of each S-MECS can dynamically adjust the range extension bias according to the network state, with the goal utility function being to maximize throughput and minimize handover costs under edge throughput constraints as follows.

[0067]

[0068]

[0069]

[0070]

[0071] Here, is the objective utility function, and , , are the set of all HMECs, the set of S-MECS, and the set of MECS beams, respectively, and is an indicator vector for the link between UE u and MECS B, and is the minimum throughput required for edge UEs.

[0072] Objective utility function is the average throughput at time t and handover costs It is defined as the sum of the weights as follows.

[0073] represents an indicator variable.

[0074] If UE u is associated with the j-th beam of MECS b =1 and otherwise =0.

[0075]

[0076] class Each represents the weight of throughput and handover cost, respectively.

[0077] is the bias vector of the MECS beams.

[0078] The hourly average throughput for all UEs is defined as follows.

[0079]

[0080] is the set of UE u.

[0081] Also, time The average handover cost for all UEs is as follows.

[0082]

[0083] Here is UE The handover for is time It is 1 if it occurs, and 0 otherwise.

[0084] In other words, average handover cost refers to the average number of handovers, and each handover may incur handover costs such as signal processing overhead and energy consumption; this cost It can be implicitly reflected in.

[0085] If a UE falls within the bottom 5% in terms of throughput, these UEs are classified as edge UEs.

[0086] In the original problem, mathematical equations 3 and 4 are Mixed Integer Problems (MIP) because they involve multiple discrete and continuous variables.

[0087] MIPs are known to be inherently NP-hard, and finding the optimal solution generally requires exponential time complexity.

[0089] Since the channels between the UE and MECS change dynamically over time, as many channels as possible can be created, which becomes a problem when applying existing optimization solutions in real time.

[0090] Therefore, we propose a new solution based on Deep RL (Reinforce Learning) to use real-time user connection control.

[0091] The following describes DQN (Deep Q-Network)-based user connection control for mm-HMEC systems.

[0092] Figure 2 is a diagram illustrating the DQN-based bias control model architecture for a mm-HMEC system.

[0093] Referring to Figure 2, several episodes are implemented during the training phase of the Deep double Q-learning algorithm.

[0094] After the state regarding the system throughput, edge UE throughput, and handover cost of the mm-HMEC environment is observed in each episode, the agent selects an action for exploration according to a greedy strategy.

[0095] Here, system throughput is the throughput of one or more S-MECS (Slave-MEC Servers) and M-MECS (Master-MEC Servers) for all UEs.

[0096] The algorithm initially prefers to search using a reasonably random policy and gradually shifts to the use of a deterministic policy.

[0097] The system performs the selected task, receives a reward, observes the next state, and the experience tuple is stored in the replay memory for training in subsequent steps.

[0099] The following describes a DQN-based centralized user connection control according to the present embodiment.

[0100] In this embodiment, to determine the bias for the optimal directional beam for the S-MECS, the M-MECS receives all information and trains the DQN model based on it.

[0101] To train the DQN model, MECS utilizes the following state, action, and reward functions.

[0102] The agent state at time t is defined as follows.

[0103]

[0104] Here, is the UE information vector assigned to S-MECS, is the UE information vector assigned to M-MECS, is the average throughput vector for edge UEs in all S-MECS at time t, is the information vector of the edge UE at time t.

[0105] The agent's behavior at time t is as follows.

[0106]

[0107] Here, time In each S-MECS It represents the set of biases for and is as follows.

[0108]

[0109] The compensation function maximizes the objective utility function, which consists of throughput and handover costs for all UEs.

[0110] In Equation 5, since there is one constraint on the throughput of the edge UE, the constraint is moved to the objective function, such as Lagrangian Relaxation, to maximize utility while satisfying the constraint.

[0111] In particular, the sigmoid function is used to reflect the minimum rate of mathematical equation 5 as the penalty term of the compensation function.

[0112]

[0113] Consequently, the compensation function is as follows.

[0114]

[0115] Here represents the weight value ( ).

[0116] After M-MECS receives state reports from all S-MECS, the state action Compensation for It is assumed that can be observed.

[0117] hour In M-MECS, the state Acting after seeing Taking and compensation Achieve.

[0118] The goal is to determine the optimal directional beam bias for all S-MECS to maximize compensation over the long term.

[0119]

[0120] Discount factor It approaches 0 when t is sufficiently high.

[0121] and are the set of edge UEs and the number of edge UEs, respectively.

[0122] FIG. 3 shows Algorithm 1 for the DQN model learning process according to the present embodiment.

[0123] Referring to Algorithm 1, M-MECS collects information on the current channel states of all S-MECS and MUEs, assembles all information into a system state, processes it, and selects an action for search according to a greedy strategy (lines 11 to 16 of Algorithm 1).

[0124] The DQN parameters are updated after performing Gradient Descent on the Training Q-value Loss with minibatch experience samples (lines 23 to 33 of Algorithm 1).

[0125] In this embodiment, experience replay is applied to improve the convergence of the DQN model, and the Q-target and - Separate exploration.

[0126] In particular, setting the bias discretely allows for a finite number of actions at a time, which can promote faster learning and convergence compared to continuously adjusting the bias.

[0128] The following describes a DQN-based centralized operation procedure according to the present embodiment.

[0129] FIG. 4 is a diagram illustrating the overall procedure for centralized bias control and handover according to the present embodiment.

[0130] Referring to Fig. 4, in a centralized control model, all UEs periodically report their measurement information to the Serving S-MECS via Radio Resource Control (RRC) messages (Measurement reports).

[0131] Here, the measurement information may include channel status, the handover rate and bit rate of each UE.

[0132] After collecting status reports from all S-MECS and M-MECS, the M-MECS executes the global DQN model according to the present embodiment to determine the bias for all beams of the system and distributes the bias to all S-MECS (Notify bias set).

[0133] The status report may include information regarding the number of all UEs and edge UEs, and the throughput of all UEs and edge UEs.

[0134] The determined bias is distributed to all S-MECS, and the S-MECS notifies SUE of its bias (Notify bias set).

[0135] Each S-MECS updates the bias.

[0136] If the UE detects a stronger pilot signal due to a bias from an adjacent MECS that persists for a given period, the UE immediately reports to its service MECS (Detect handover).

[0137] To perform a handover, the service MECS makes a handover decision and sends a request to the target MECS so that a new downlink resource can be allocated between the UE and the target MECS.

[0138] The service MECS also transfers the UE's state to the target MECS for continuous data communication between the UE and the target MECS (Status transfer).

[0139] Subsequently, a handover is performed between the UE and the target MECS.

[0141] The DQN-based distributed user connection control according to the present embodiment is described below.

[0142] In distributed user connection control, all S-MECS independently determine their own biases using a small amount of information exchange between S-MECS and M-MECS.

[0143] State, behavior, and reward differ as follows compared to centralized user connection control.

[0144]

[0146] FIG. 5 is a diagram illustrating a procedure for DQN-based distributed bias control and handover according to the present embodiment.

[0147] Referring to Figure 5, in the distributed control model, all UEs periodically report their measured information to the service MECS via RRC messages.

[0148] Unlike the centralized model, M-MECS transmits global status information, such as overall system throughput, to the service S-MECS (Notify system status).

[0149] Each S-MECS receives global and local information from M-MECS and service SUE, respectively, then uses a local DQN learning module to determine the bias for the beam and updates the bias (Update bias).

[0150] Subsequently, S-MECS shares the determined bias with other S-MECS through M-MECS (Notify bias set), and all UEs can know the biases of all adjacent MECS through the MECS broadcast procedure.

[0151] Then, the user connection procedure is the same as the centralized model.

[0153] To demonstrate the effectiveness of the invention, the developed centralized operation technique and distributed operation procedure were compared with the existing techniques described below.

[0154] Random CRE (R-CRE): Random beam bias

[0155] Standard CRE (S-CRE): As a static CRE system, the static bias is set to 5.

[0156] Dynamic CRE (D-CRE): As a dynamic CRE scheme, it adjusts the bias based on the resource balance between M-MECS and S-MECS. If throughput decreases, the bias increases by 1 dB, otherwise it decreases by 1 dB.

[0157] Q-learning based CRE (Q-CRE): As an RL-based dynamic CRE scheme, it performs Q-learning based bias control. All M / S-MECS learn their own biases.

[0159] Figure 6 is a diagram illustrating the system utility of an arbitrary destination-based mobility model.

[0160] Referring to Figure 6, the system utility of the random destination-based mobility model provides 25.13–83.85% higher utility based on 4 MECS standards and 41.49–151.37% higher utility based on 10 MECS standards.

[0161] Figure 7 is a diagram illustrating the system utility of an arbitrary direction-based mobility model.

[0162] Referring to Figure 7, the system utility of the random direction-based mobility model provides 33.18–83.85% higher utility based on 4 MECS standards and 41.49–129.65% higher utility based on 10 MECS standards.

[0163] Figure 8 is a diagram illustrating a Manhattan mobility model-based system utility.

[0164] Referring to Figure 8, the Manhattan Mobility Model-based System Utility provides 19.66–103.5% higher utility based on 4 MECS standards and 40.1–277.44% higher utility based on 10 MECS standards.

[0165] FIG. 9 is a diagram illustrating the cumulative distribution function of throughput in an arbitrary destination-based mobility model, FIG. 10 is a diagram illustrating the cumulative distribution function of throughput in an arbitrary direction-based mobility model, and FIG. 11 is a diagram illustrating the cumulative distribution function of throughput in a Manhattan mobility model-based system utility.

[0166] Referring to Figures 9 through 11, each model-based system utility provides 15.32–63.87% higher throughput at 4 MECS criteria and 10 MECS criteria.

[0167] FIG. 12 is a diagram showing the number of handovers in an arbitrary destination-based mobility model, FIG. 13 is a diagram showing the number of handovers in an arbitrary direction-based mobility model, and FIG. 14 is a diagram showing the number of handovers in a Manhattan mobility model.

[0168] Referring to Figures 12 to 14, it can be seen that the number of handovers for the three mobility models also decreased by at least 25.4% and up to 71.52% compared to the control group.

[0170] The DQN-based user connection control method of a hierarchical mobile edge computing system for the aforementioned mobile IoT service may also be implemented in the form of a recording medium containing computer-executable instructions, such as applications or program modules executed by a computer. A computer-readable medium may be any available medium accessible by a computer and includes both volatile and non-volatile media, as well as removable and inseparable media. Additionally, a computer-readable medium may include a computer storage medium. A computer storage medium includes both volatile and non-volatile, removable and inseparable media implemented by any method or technique for storing information, such as computer-readable instructions, data structures, program modules, or other data.

[0171] The DQN-based user connection control method of the layered mobile edge computing system for the aforementioned mobile IoT service may be executed by an application that is basically installed on a computing device (which may include a program included in a platform or operating system installed on the computing device), or by an application (i.e., a program) that is directly installed by a user on a master terminal through an application provider server, such as an application store server, an application, or a web server related to the service. In this sense, the DQN-based user connection control method of the layered mobile edge computing system for the aforementioned mobile IoT service may be implemented as an application (i.e., a program) that is basically installed on a computing device or directly installed by a user, and may be recorded on a computer-readable recording medium such as a terminal.

[0172] The embodiments of the present invention described above are disclosed for illustrative purposes only, and those skilled in the art with ordinary knowledge of the present invention may make various modifications, changes, and additions within the spirit and scope of the present invention, and such modifications, changes, and additions should be considered to fall within the scope of the following claims.

Claims

Claim 1 As a Deep Q-learning Network (DQN)-based user connection control device for a Hierarchical Mobile Edge Computing (HMEC) system for mobile IoT services, a processor; and includes memory connected to the processor, wherein the memory stores program instructions executed by the processor to select an action to determine a beam-specific range extension bias of each MECS that maximizes rewards from the collected state, wherein the memory collects a state including at least one of the throughput of one or more S-MECS (Slave-MEC Server) and M-MECS (Master-MEC Server) for all UEs from the mm-HMEC environment, the throughput of an edge UE whose throughput is less than or equal to a preset value, and the handover cost, and wherein each of the one or more S-MECS and M-MECS notifies the UE connected to it of the bias determined corresponding to each, and if the first UE detects a pilot signal stronger than the previously connected service MECS due to the determined bias from the target MECS to be handed over, it reports to its service MECS, and the service S-MECS to which the first UE is connected and receiving services reports to the target MECS so that new downlink resources can be allocated between the first UE and the target MECS for handover execution. A user connection control device that transmits a request, wherein the service MECS transmits the state of the first UE to the target MECS for continuous data communication between the first UE and the target MECS, the state comprising a UE information vector assigned to one or more S-MECSs, a UE information vector assigned to the M-MECS, an average throughput vector for edge UEs in all S-MECSs at time t, and an information vector of edge UEs at time t. Claim 2 In claim 1, the user connection control device is an M-MECS connected to the one or more S-MECSs, and the program instructions are a user connection control device that determines the bias of each of the one or more S-MECSs based on the DQN model and distributes the determined bias to the one or more S-MECSs. Claim 3 In claim 1, the user connection control device is an S-MECS connected to the M-MECS, and each S-MECS receives measurement information from a UE connected to it, transmits the measurement information to the M-MECS, and receives the state from the M-MECS to determine its own bias. Claim 4 delete Claim 5 In paragraph 2, the DQN model is a user connection control device that determines the bias according to the following mathematical formula.[Mathematical Formula] Objective utility function is time It is defined as the weighted sum of the average throughput of all UEs, and , , are the set of all HMECs, the set of S-MECS, and the set of MECS beams, respectively, and is an indicator vector for the link between UE u and MECS B, and is the minimum throughput required for the edge UE, and the program instructions are a user connection control device that determines a bias that maximizes the objective utility function. Claim 6 In paragraph 5, the weighted sum of the above average throughput is defined by the following mathematical formula, [mathematical formula] class are the weights for throughput and handover cost, respectively, and is the bias vector of the MECS beams. The hourly average throughput for all UEs is defined by the following mathematical formula, and [Equation] is the set time of UE u The average handover cost for all UEs in the User Connection Control Unit is defined by the following mathematical formula. [Mathematical Formula] Here is UE The handover for is time It is 1 if it occurs, and 0 otherwise. Claim 7 delete Claim 8 A Deep Q-learning Network (DQN)-based user connection control system for a Hierarchical Mobile Edge Computing (HMEC) system for mobile IoT services, comprising a Master-MEC Server (M-MECS) that collects a state from an mm-HMEC environment including at least one of the throughput for all UEs, the throughput of edge UEs whose throughput is less than or equal to a preset value, and the handover cost;It includes one or more S-MECS (Slave-MEC Servers) that receive measurement information from one or more UEs connected to themselves, assemble it into a state in a DQN model, and transmit it to the M-MECS, wherein the M-MECS selects an action to determine a beam-specific range extension bias of each MECS that maximizes the reward from the collected state based on a DQN (Deep Q-learning Network) model, whether the M-MECS is centrally or each of the one or more S-MECSs is distributed; each of the one or more S-MECSs and M-MECS notifies the UEs connected to themselves of the bias determined accordingly; if the first UE detects a pilot signal stronger than the previously connected service MECS due to the determined bias from the target MECS to be handed over, it reports to its service MECS; the service S-MECS to which the first UE is connected and receiving services transmits a request to the target MECS so that a new downlink resource can be allocated between the first UE and the target MECS for the handover to be performed; and the service MECS [requires] continuous data between the first UE and the target MECS A user connection control system that transmits the state of the first UE to the target MECS for communication, wherein the state includes a UE information vector assigned to one or more S-MECSs, a UE information vector assigned to the M-MECS, an average throughput vector for edge UEs in all S-MECSs at time t, and an information vector of edge UEs at time t. Claim 9 A user connection control system according to claim 8, wherein when the M-MECS determines the bias, the M-MECS determines the bias of each of the one or more S-MECS based on the DQN model and distributes the determined bias to the one or more S-MECS. Claim 10 A user connection control system according to claim 8, wherein each of the above one or more S-MESCs determines the bias, each S-MECS receives measurement information from a UE connected to itself, transmits the measurement information to the M-MECS, and receives the state from the M-MECS to determine its own bias. Claim 11 delete Claim 12 A method for controlling a Deep Q-learning Network (DQN)-based user connection of a Hierarchical Mobile Edge Computing (HMEC) system for mobile IoT services in a device including a processor and memory, comprising the step of collecting a state including at least one of the throughput of one or more Slave-MEC Servers (S-MECS) and Master-MEC Servers (M-MECS) for all UEs from an mm-HMEC environment, the throughput of an edge UE whose throughput is less than or equal to a preset value, and a handover cost; The method includes the step of selecting an action to determine a beam-specific range expansion bias of each MECS that maximizes the reward from the collected state based on a Deep Q-learning Network (DQN) model, wherein each of the one or more S-MECSs and M-MECSs notifies the UE connected to it of the bias determined correspondingly, and if the first UE detects a pilot signal stronger than the previously connected service MECS due to the determined bias from the target MECS to be handed over, it reports to its service MECS, and the service S-MECS to which the first UE is connected and receiving service transmits a request to the target MECS so that new downlink resources can be allocated between the first UE and the target MECS for performing a handover, and the service MECS transmits the state of the first UE to the target MECS for continuous data communication between the first UE and the target MECS, wherein the state comprises a UE information vector assigned to the one or more S-MECSs, a UE information vector assigned to the M-MECS, an average throughput vector for edge UEs in all S-MECSs at time t, and time A user connection control method including an information vector of an edge UE at t.

Images