Ultra-dense heterogeneous Internet of Things resource allocation method and device based on undirected graph and WCSAC

Abstract

The invention discloses an ultra-dense heterogeneous Internet of Things resource allocation method and device based on an undirected graph and WCSAC. The method comprises the following steps: constructing a communication network architecture in an HUD-IoT scene; establishing a conflict graph used for representing the conflict condition of the access points in the communication network architecture, and constructing a conflict-free graph based on the conflict graph; establishing a partial order relationship on the conflict-free graph of the communication network architecture, and constructing a Haas graph based on the partial order relationship; a conflict-free resource reuse optimization problem of a communication network architecture is established, a WCSAC reinforcement learning algorithm is adopted, and an intelligent agent is trained under a constrained Markov decision process framework, so that a final maximum conflict-free node set is obtained through the trained intelligent agent based on a Haas diagram output action strategy aiming at the optimization problem. And conflict-free resource reuse in the HUD-IoT scene is realized. According to the method, the maximum conflict-free node set can be output, so that resource multiplexing conflicts can be effectively relieved.

Description

Technical Field

[0001] This invention relates to the field of ultra-dense Internet of Things (IoT) technology, and in particular to an ultra-dense heterogeneous IoT resource allocation method and apparatus based on undirected graphs and WCSAC. Background Technology

[0002] To address the growing wireless communication demands of the Internet of Things (IoT), smart network devices and wireless services are forming a broad and increasingly complex landscape. The IoT is experiencing exponential growth, leading to a surge in demand for massive access and high-capacity transmission. To improve network capacity and data transmission rates, communication networks are evolving into ultra-dense heterogeneous IoT (HUD-IoT) with massive overlapping coverage. This will be accompanied by an increase in the number of base stations (BSs) and the deployment of high-density access nodes. Compared to traditional network architectures, HUD-IoT is characterized by high heterogeneous node density and extensive resource area overlap, leading to severe channel interference and resource reuse conflicts, thus increasing the complexity of network management. The current key challenge lies in developing effective resource allocation optimization strategies.

[0003] Existing research on mitigating network management complexity using resource reuse techniques mainly focuses on improving data transmission rates, reducing inter-cell interference, optimizing power control, and improving energy efficiency. While these studies have made significant progress, they still fail to fully consider co-channel interference caused by heterogeneous protocols and coverage overlap issues among numerous devices. To overcome these limitations, a HUD-IoT resource allocation strategy is needed to effectively mitigate resource reuse conflicts. Summary of the Invention

[0004] This invention aims to at least partially address the technical problems in related technologies. Therefore, the first objective of this invention is to provide a method for allocating ultra-dense heterogeneous IoT resources based on undirected graphs and WCSAC, which can output the maximum set of conflict-free nodes, thereby effectively mitigating resource reuse conflicts.

[0005] The second objective of this invention is to provide an ultra-dense heterogeneous Internet of Things resource allocation device based on undirected graphs and WCSAC.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0007] A method for allocating ultra-dense heterogeneous IoT resources based on undirected graphs and WCSAC includes:

[0008] Construct a communication network architecture for HUD-IoT scenarios, which includes multiple base stations, access points, and IoT devices;

[0009] A conflict graph is established to represent the access point conflict situation in the communication network architecture, and a conflict-free graph is constructed based on the conflict graph;

[0010] A partial order relation is established on the conflict-free graph of the communication network architecture, and a Hasse graph is constructed based on the partial order relation to represent the linear chain structure of the maximum conflict-free node set.

[0011] The conflict-free resource reuse optimization problem of the aforementioned communication network architecture is established. The WCSAC reinforcement learning algorithm is used to train an agent under the constrained Markov decision process framework. The trained agent outputs an action policy based on the HASS diagram to obtain the final maximum set of conflict-free nodes, thereby realizing conflict-free resource reuse in the HUD-IoT scenario. The action policy is to select two access points or two maximum sets of conflict-free nodes for merging.

[0012] In one possible implementation, the coverage areas of base stations and access points are determined, and a conflict graph is established based on the coverage areas and Euclidean distances. The adjacency matrix of the conflict graph satisfies:

[0013]

[0014] Among them, M adj(i,j) D represents the adjacency matrix of the conflict graph. i,j Let l represent the Euclidean distance between the i-th and j-th access points. i l j Let represent the coverage radii of the i-th and j-th access points, respectively.

[0015] In one possible implementation, access points that can share the channel without interference are determined based on the conflict graph to construct a conflict-free graph and obtain the adjacency matrix of the conflict-free graph.

[0016] In one possible implementation, establishing a partial order relation includes:

[0017] For any set of access points A, let the partial order relation be...

[0018] If for all If a partial order relation R satisfies antisymmetry and transitivity, then the partial order relation R is determined to be a partial order relation on the set of access points A, and the ordered pairs (R, A) constitute a partial order set, where x, y, and z are the first to third access points, respectively.

[0019] In one possible implementation, constructing the Hasse diagram includes:

[0020] Node pairs that satisfy transitivity and antisymmetry are selected based on partial order relations;

[0021] The node pairs are filtered to select the edges that need to be retained and to remove redundant edges.

[0022] The linear chain structure is formed based on the retained node pairs and edges;

[0023] Multiple linear chain structures are merged to construct the Hasse diagram.

[0024] In one possible implementation, the conflict-free resource reuse optimization problem is to identify and output the maximum conflict-free node set to maximize channel reuse efficiency while minimizing the maximum number of conflict-free nodes.

[0025] In one possible implementation, the WCSAC reinforcement learning algorithm is used to train the agent within a constrained Markov decision process framework, including:

[0026] The state space is defined as the partial order relationship between nodes, and the action space is defined as the node merging operation. The agent is trained, and an entropy regularization mechanism is used to balance the action policy. Two safety evaluation networks are used to evaluate the safety of the action policy. The two safety evaluation networks are the mean network and the variance network, respectively. The two safety evaluation networks are updated by the Bellman operator.

[0027] In one possible implementation, during agent training, a reward function is used to incentivize the agent to reduce the number of the maximum conflict-free node set based on a non-linear incentive mechanism; a cost function is used to apply negative feedback to merging actions that lead to conflicts, so as to ensure interference-free communication.

[0028] In one possible implementation, the final output set of maximum conflict-free nodes satisfies the following conditions: it covers all access points; there are no conflicts between access points within each maximum conflict-free node set; and there are no channel multiplexing conflicts between different maximum conflict-free node sets.

[0029] To achieve the above objectives, a second aspect of the present invention provides an ultra-dense heterogeneous Internet of Things resource allocation device based on undirected graphs and WCSAC, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can implement the method described above.

[0030] This invention has at least the following technical effects:

[0031] This invention provides a resource allocation method for ultra-dense heterogeneous IoT based on undirected graphs and WCSAC. The proposed algorithm employs a reinforcement learning framework, autonomously generating conflict-free solutions through self-learning. Unlike traditional methods that require pre-setting initial states, this method utilizes global observation of the solution space, eliminating the dependency on initialization. By modeling the task as a restricted Markov decision process, the algorithm naturally achieves no initialization operation. Furthermore, the agent's optimization trajectory is guided by a carefully designed reward function, which penalizes conflicts and prioritizes feasible solutions. Additionally, this invention's conflict-free resource reuse method for HUD-IoT scenarios can cover as many nodes as possible with the maximum conflict-free node set, effectively reducing the required resource volume while ensuring conflict-free resource reuse.

[0032] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0033] Figure 1 This is a flowchart of a method for allocating ultra-dense heterogeneous IoT resources based on undirected graphs and WCSAC, according to an embodiment of the present invention.

[0034] Figure 2 This is a local conflict relationship diagram in the HUD-IoT scenario of this invention embodiment.

[0035] Figure 3 This is a conflict graph of the access point constructed according to an embodiment of the present invention.

[0036] Figure 4 This is a description of the structural framework and flowchart of the weight update mechanism in the algorithm of this invention.

[0037] Figure 5 This is a schematic diagram illustrating the changes in cumulative reward during the training process according to an embodiment of the present invention.

[0038] Figure 6 This is a schematic diagram illustrating the changes in cumulative cost during the training process according to an embodiment of the present invention.

[0039] Figure 7 This is a schematic diagram comparing the performance of different algorithms in an embodiment of the present invention.

[0040] Figure 8 This is a schematic diagram illustrating the performance of an embodiment of the present invention under the condition of increased edge density. Detailed Implementation

[0041] The following describes this embodiment in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the invention, and should not be construed as limiting the invention.

[0042] The following description, with reference to the accompanying drawings, illustrates a method and apparatus for allocating ultra-dense heterogeneous Internet of Things resources based on undirected graphs and WCSAC.

[0043] Figure 1 This is a flowchart illustrating a method for allocating ultra-dense heterogeneous IoT resources based on undirected graphs and WCSAC, according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0044] Step S101: Construct a communication network architecture for the HUD-IoT scenario, which includes multiple base stations, access points, and IoT devices.

[0045] In this embodiment, the communication network architecture in the HUD-IoT scenario consists of multiple base stations (BS), access points (AP), and a large number of IoT devices. Base stations and access points provide signal coverage for IoT devices, while access points also relay information by communicating with base stations or other access points. In this scenario, the primary focus is on the communication quality of IoT devices, especially signal interference, and a centralized resource allocation scheme is considered for mitigation.

[0046] Specifically, there are N base stations, denoted as V. b ={BS1,BS2,…,BS N}, and M access points, denoted as V a ={AP1,AP2,…,AP M}, where V b For base station set, BS N For the Nth base station, V a For the set of access points, AP M This is the Mth access point. The base station serves as the network communication interface between the access point and IoT devices, while the access point acts as a relay for the base station signal to extend communication coverage. IoT devices are typically responsible for collecting their own location information and receiving command messages to respond accordingly. They communicate with the base station or access point by sending and receiving signals, transmitting data to a cloud server connected via the internet.

[0047] There are numerous and widely distributed IoT devices. To achieve broad coverage of these devices, base stations and access points are deployed in the network. Access points act as relays, forwarding signals from base stations. For IoT devices, both base stations and access points are considered network service providers. When IoT devices communicate through the network, they often receive signals on the same frequency from multiple base stations or access points, causing signal interference and leading to communication failures.

[0048] IoT devices should possess real-time information acquisition and control capabilities, which typically rely on stable network connections. Due to cost constraints, IoT devices often lack high-capacity communication capabilities. To ensure communication stability, access points are needed to extend coverage and maintain high-quality signals for IoT devices. Even in this scenario, due to co-channel interference, IoT devices may still not achieve ideal signal quality.

[0049] Signal-to-interference-plus-noise ratio (SINR) is a commonly used metric for measuring signal quality. When IoT devices are located in overlapping coverage areas of multiple access points using the same channel resources, co-channel interference can cause a decrease in SINR.

[0050] Specifically, the set of communication nodes can be defined as V, which includes base stations and access points, i.e., V = {…BS} N AP M The signal quality SINR of IoT devices can be expressed as shown in formula (1):

[0051]

[0052] Among them, SINR s’ V represents the signal quality of IoT devices under channel resources s'. i1 V j1 These represent the i1th or j1th base station or access point in the communication node set V that communicates with the IoT device. Representing communication node V respectively i1 V j1 Transmit power on the same channel resources, Representing communication node V respectively i1 V j1 The channel gain is N0, and the noise power is N0.

[0053] When IoT devices are at communication node V i1 V j1 When covering overlapping areas, This can increase simultaneously, leading to increased interference and thus reducing the signal's SINR value. Similarly, communication between access points can also be affected by overlapping coverage areas, causing interference.

[0054] Step S102: Establish a conflict graph to represent the access point conflict situation in the communication network architecture, and construct a conflict-free graph based on the conflict graph.

[0055] Under the condition of fixed transmission power, reduce Or change the communication node V j1 Channel resources, as shown in formula (1), are an effective method to improve SINR. Since the locations of the access point and base station are fixed, changing the communication node V... j1 Channel resources are an effective means to improve SINR. To avoid using co-channel resources in overlapping coverage areas, APs and BSs should allocate channel resources reasonably.

[0056] Channel gain is affected by the distance between nodes. Therefore, taking the access point as an example, we define the Euclidean distance l. i As a corresponding The coverage radius, where l i Indicates access point V i The coverage radius, Indicates access point V i The channel gain. To ensure that the SINR exceeds the threshold, i.e., the SINR limit of IoT devices, when access point V... i and V j When using the same channel resources, the Euclidean distance D between them i,j D must be satisfied i,j >l i +l j Among them, D i,j It is expressed as follows:

[0057]

[0058] Where, x i y i Access point V i x and y coordinates j y j Access point V j The horizontal and vertical coordinates.

[0059] Therefore, an undirected graph, i.e., a conflict graph G, can be constructed. c To represent conflict situations in HUD-IoT scenarios. Conflict graph G c The adjacency matrix is ​​shown in formula (3).

[0060]

[0061] Among them, M adj(i,j) The adjacency matrix of the conflict graph, l j Indicates access point V j The coverage radius.

[0062] like Figure 2 As shown, this illustrates the local conflict relationships in a HUD-IoT scenario. Figure 2 There are four access points (APs) covering the area, where l3 represents the coverage radius of AP3, and D... 2,3 This represents the Euclidean distance between AP2 and AP3. For example... Figure 2 As shown, due to D 2,3 <l2+l3, the coverage areas of AP2 and AP3 overlap, affecting devices 2 and 3 in the HUD-IoT. Device 1 is also affected by the overlapping coverage of AP1 and AP2. In addition, there is also a conflict between AP4 and AP3.

[0063] In real-world scenarios, the coverage area of ​​an access point (AP) may vary depending on its orientation, therefore... i It is not a fixed value. The wall between AP1 and AP2 causes signal shadowing, which in turn changes the coverage radius, but coverage overlap still exists, so M in the adjacency matrix... adj(1,2) It should be set to 1.

[0064] according to Figure 2 The conflict graph G of the access points can be constructed based on formula (3). c and as Figure 3 The following is a demonstration. It should be noted that the two types of interference involving access points and IoT devices are as follows:

[0065] (1) Definition 1: When access point V i and V j Using the same channel resources and adjacency matrix M adj(i,j) When = 1, interference will occur. For IoT devices, this will cause a decrease in SINR, as shown in formula (1). For access points, communication conflicts will occur due to co-channel conflicts.

[0066] (2) Definition 2: When access point V i and V j Using the same channel resources and satisfying Interference may still occur at that time. This represents the two-hop adjacency matrix. In this case, it originates from access point V. j The transmission signal may reduce the access point V i The SINR of direct neighbors is as described by formula (1).

[0067] To ensure communication quality, the interference situations described in Definitions 1 and 2 must be avoided when access points reuse channel resources. Furthermore, reducing the number of shared channel resources helps support higher-density HUD-IoT deployments and expands overall coverage.

[0068] To address the interference issues described in Definitions 1 and 2, this embodiment proposes constructing a partially ordered set to model conflict relationships and employing the WCSAC (Secure Reinforcement Learning) algorithm to identify the maximally conflict-free node set (MCNS).

[0069] In the HUD-IoT scenario, the set of access points is defined as D = {V i V j ,…V k} represents a group of access points that reuse the same channel resource, where V k Let $k$ represent the k-th access point. All access points (APs) in the set $D$ operate on the same channel. A set ${D1, D2, ... D$ is defined. K} It covers all APs in the HUD-IoT scenario, where D K Let K represent the set of access points, where each set of access points contains access points (APs) sharing the same channel. Furthermore, there are no conflicts between any two sets of access points; that is, they are conflict-free.

[0070] To achieve conflict-free grouping with the minimum number of sets, this embodiment introduces the concept of a maximum conflict-free node set to represent each access point set D. Multiple MCNSs are constructed to cover all access points; the formal definition of an MCNS is shown in equation (4).

[0071]

[0072] Here, x1 and x2 represent two different access points belonging to the access point set D, and C(x1) represents the set of access points that conflict with access point x1. For any access point v∈VD, if D∪v is not a conflict-free node set, then the access point set D is called a maximal conflict-free node set. In HUD-IoT scenarios, there is a resource allocation problem with multiple coverages, which can be solved by identifying multiple MCNSs that jointly cover all nodes in the communication node set V. It is understood that the above only uses access points as an example, and its essence can also be extended to communication node base stations; therefore, all nodes mentioned also include base stations. Of course, all the following content will still be explained using access points as an example.

[0073] To achieve conflict-free channel resource reuse and reduce management complexity in HUD-IoT scenarios, finding the minimum number of the largest conflict-free node set to cover all access points is an NP-hard problem.

[0074] Existing methods for maximizing the set of conflict-free nodes typically rely on exhaustive search, resulting in high computational complexity. This embodiment introduces a conflict-free graph G. cf Let G represent the conflict-free relationships between APs, where the conflict-free graph G is... cf Through the conflict graph G c The resulting conflict-free graph is the complement of the conflict graph, obtained by subtracting the conflict graph from the fully connected graph. Based on this graph, the corresponding adjacency matrix, i.e., the conflict-free adjacency matrix M, can be derived. cf It is used to identify AP nodes that can share the same channel resources. The WCSAC algorithm operates on a collision-free graph G with a partial order relation. cf Multiple linear chains are extracted from them, and these linear chains are used as MCNS.

[0075] Step S103: Establish a partial order relation on the conflict-free graph of the communication network architecture, and construct a Hasse graph based on the partial order relation to represent the linear chain structure of the maximum conflict-free node set.

[0076] Establishing a partial order relation includes: for any set of access points A1, let the partial order relation... If for all If the partial order relation R satisfies antisymmetry and transitivity, i.e., satisfies formulas (5) and (6), then the partial order relation R is determined to be a partial order relation on the set of access points A1, and the ordered pairs (R, A1) constitute a partial order set, where x, y, and z are the first to third access points, respectively.

[0077]

[0078] in, This indicates the complement relation of R, that is, the relation that does not belong to R.

[0079] If R = G cf For any subset, i.e., the set of access points The constructed partial order relation structure<A1,R> It satisfies the following properties:

[0080] Transitivity: All access points satisfy the transitivity relation, that is:

[0081]

[0082] Interconnectivity: Any pair of access points can be interconnected, that is:

[0083]

[0084] Therefore, the partial order relation structure<A1,R> A total ordered subset is formed under the partial order relation R.

[0085] In one possible implementation, constructing a Hasse graph includes: filtering node pairs that satisfy transitivity and antisymmetry based on partial order relations; filtering edges that need to be retained based on the selected node pairs and removing redundant edges; forming a linear chain structure based on the retained node pairs and edges; and merging multiple linear chain structures to construct the Hasse graph.

[0086] Hasse diagrams are a concise and intuitive graphical representation of partially ordered sets. A partially ordered set is a set equipped with binary relations (satisfying antisymmetry and transitivity, with reflexivity usually omitted in the diagram). In the context of partially ordered sets, Hasse diagrams demonstrate the partially ordered structure by visualizing the overlapping relationships between elements, while omitting edges derived from transitivity to reduce visual complexity.

[0087] Specifically, when element a ≤ b (another element) and there is no intermediate element c such that a ≤ c ≤ b, a Hasse diagram draws an edge from a to b (usually upwards). By omitting reflexive and transitive relations, the Hasse diagram captures the core hierarchical structure of the partial order in the most concise and readable form.

[0088] Hasse diagrams, as a simplified relational graph, can represent the maximum set of conflict-free nodes as a linear chain, where each pair of adjacent elements satisfies the conflict-free condition. By identifying the longest linear chain in the Hasse diagram, a maximum-consolidation network node (MCNS) can be efficiently determined, thus significantly reducing the computational complexity of the algorithm.

[0089] In a Hasse graph, the maximally conflict-free set of nodes (MCNS) is represented by a single linear chain, which essentially avoids the need to search for any subset. Partial order relations reduce unnecessary edges during Hasse graph construction, including cycles and edges implied by the transitivity of access points.

[0090]

[0091] Among them, D i2 and D j2 All are solutions with the maximum set of conflict-free nodes, and represent repeated results. In the conflict-free graph G... cf Derived partial order relation structure<A1,R> In the algorithm, only the longest path from the minimum element to the maximum element is retained.

[0092]

[0093] Among them, P s Representing the partial order relation structure<A1,R> P is the set of paths from the smallest element to the largest element. i3 P j3These represent two different paths. Since these paths do not contain each other, selecting the longest path yields the maximum solution for the maximum conflict-free node set (MCNS). Therefore, redundant edges can be removed using the aforementioned partial order relation, resulting in a linear chain structure, thus obtaining a Hasse graph. Selecting the longest path from the Hasse graph yields the maximum solution for the maximum conflict-free node set (MCNS).

[0094] Step S104: Establish a conflict-free resource reuse optimization problem for the communication network architecture. Use the WCSAC reinforcement learning algorithm to train an agent under the constrained Markov decision process framework. The trained agent outputs an action policy based on the HASS diagram to solve the conflict-free resource reuse optimization problem, so as to obtain the final maximum set of conflict-free nodes and realize conflict-free resource reuse in the HUD-IoT scenario. The action policy is to select two access points or two maximum sets of conflict-free nodes for merging.

[0095] The conflict-free resource reuse optimization problem aims to identify and output the maximum conflict-free node set to maximize channel reuse efficiency while minimizing the number of maximum conflict-free node sets. The final output maximum conflict-free node set satisfies the following conditions: it covers all access points; there are no conflicts between access points within each maximum conflict-free node set; and there are no channel reuse conflicts between different maximum conflict-free node sets.

[0096] In this embodiment, the WCSAC reinforcement learning algorithm is used to train the agent within the constrained Markov decision process framework. This includes: defining the state space as the partial order relationship between nodes and the action space as the node merging operation; training the agent; using an entropy regularization mechanism to balance the action policy; and using two safety evaluation networks to evaluate the safety of the action policy. The two safety evaluation networks are a mean network and a variance network, respectively. The two safety evaluation networks are updated using the Bellman operator.

[0097] It should be noted that when training the agent, a reward function can be used to incentivize the agent to reduce the number of the maximum number of conflict-free nodes based on a non-linear incentive mechanism; a cost function can be used to apply negative feedback to the merging actions that cause conflicts, so as to ensure interference-free communication.

[0098] Specifically, reinforcement learning is used in conflict-free graphs G cf The algorithm seeks the largest conflict-free set of nodes, known as the MCNS. In each step, reinforcement learning selects an access point to add to the set as the current action, aiming to maximize the cumulative reward. However, reinforcement learning struggles to fully capture the conflict relationships between the current action and previous actions. To address this, partial order relations and Hasse diagrams are introduced to guide the reinforcement learning algorithm, and the WCSAC algorithm is deployed to avoid conflicts.

[0099] Deploying Hasse diagrams in the WCSAC algorithm, such as Figure 4 As shown, the WCSAC algorithm selects one access point to add to the set at each step and constructs a Hasse graph based on the partial order relation. According to the Hasse graph, the WCSAC algorithm selects the next access point to add to the existing partial order set, which can be represented as a maximally conflict-free set (MCNS).

[0100] At this point, the optimization objective can be expressed as Equation (11). The optimization objective is to minimize the total number of MCNSs while identifying the largest partial-order chain, i.e., the linear chain, as the largest collision-free node set. Among them, maximizing the partial-order chain ensures that the channel multiplexing efficiency is maximized.

[0101]

[0102] Minimal N1 This represents the minimum number of MCNSs, i.e., the minimum number of resources used. N1 represents the number of MCNSs, and D... h This represents the h-th MCNS in the obtained MCNS, where h represents the MCNS label, V represents the set of communication nodes, Define1=0 indicates that there is no conflict in Conflict Definition 1, and Define2=0 indicates that there is no conflict in Conflict Definition 2.

[0103] The coverage overlap problem in HUD-IoT, also known as the optimization problem mentioned above, can be solved by identifying or outputting multiple Maximum Collision-Free Node Sets (MCNS). These MCNS should collectively cover all nodes (in this example, all access points), while minimizing the total number of MCNS. Furthermore, there should be no collisions between access points within each MCNS, and no channel multiplexing collisions between different MCNS.

[0104] To find the largest set of conflict-free nodes, a constrained Markov decision process (CMDP) environment for WCSAC can be constructed and trained to find the largest set of conflict-free nodes.

[0105] First, the agent combining the WCSAC algorithm and the CMDP framework is trained. During training, the state space is first defined, divided into the initial state without initial ordering relation (equation (14)) and the process state space containing ordering relation (equation (16) in the second row). The state space represents the partial ordering relation between all access points. Each access point ID (identity identifier) ​​V i Mapped to access point number O j Used in access point sequence number O j Construct a partial order relation R. Let t be the index of the access point number, then the partial order relation R can be constructed according to formula (12).

[0106]

[0107] Among them, O t These are nodes in a pseudo-order relation sequence.

[0108] In the initial state space, the access point V i There is no order relationship between them. The partial order relationship R can be determined based on the conflict-free graph G. cf The construction is as shown in formula (13):

[0109] R init =G cf (13)

[0110] From the initial partial order relation S init Construct the initial state R init :

[0111] S init =R init ×R init T (14)

[0112] When action policy π selects an action (i.e., merging two access points or two maximum conflict-free node sets MCNS), the operation is formally expressed as follows:

[0113] V k →O t D = {O1,…,O} t-1 O t}={V i V j ,…,V k ,} (15)

[0114] Among them, V k This represents the k-th access point.

[0115] Once an order relationship is established between access points, the partial order relationship R will be updated, and the state will also be updated according to the new partial order relationship.

[0116]

[0117] S = R × R T *C

[0118] Where S represents the state space, C represents the constraint matrix, and C is used to shield access points that cannot form the maximum set of conflict-free nodes.

[0119]

[0120] If the set of access points D and access point V j If a partially ordered set cannot be formed, then the access point V... jActions that will be excluded from the state space (i.e., not included in the current state's candidate actions), C i,j This represents the masking relationship between nodes i and j.

[0121] Further defining the action space. During the merging of multiple MCNSs, each step selects two access points or MCNSs to merge. To simplify the representation of the action space, when two MCNSs are merged into one MCNS, the new MCNS inherits the labels of its constituent access points.

[0122] Therefore, the action space can be represented as an upper triangular matrix, in which there is only one non-zero element at any given time, corresponding to the node pair currently selected for merging:

[0123]

[0124] When A i,j When = 1, as shown in formula (18), it indicates that the MCNS or access points labeled i and j are merged. Therefore, the agent can be trained using the state space and action space defined above.

[0125] When training the agent, a reward function can be used to incentivize the agent to reduce the number of the maximum number of conflict-free nodes based on a nonlinear incentive mechanism; a cost function can be used to apply negative feedback to the merging action that causes conflict, so as to ensure interference-free communication.

[0126] Specifically, the reward indicates whether the action taken by the agent is appropriate. In the WCSAC algorithm, this evaluation is divided into two parts: a reward function and a cost function. The reward function is used to incentivize the agent to find high-scoring solutions in the environment. In this case, as shown in Equation (19), a nonlinear reward or incentive mechanism can effectively incentivize the agent to take actions that lead to longer sequences, i.e., outputting a linear chain of longer sequences. The reward designed in this embodiment also reflects the number of MCNS, incentivizing the agent to minimize this number. On the other hand, the cost function represents ineffective actions, which is used to guide the agent to avoid conflicting merging actions. The reward function and cost function can be expressed as Equation (19).

[0127] Merging operations within the action space are rewarded when the merging conditions are met. Conversely, if a merge results in a conflict that invalidates the MCNS, it is penalized. This reward mechanism incentivizes agents to maximize the number of successful merges while effectively avoiding conflicting merge actions.

[0128]

[0129] Where r(s,a) represents the reward function for the current action-state pair, s represents the state, a represents the action, k1 represents the hyperparameters during training, N2 represents the total number of communication nodes, and M...e Let M represent the e-th action space. j Let f represent the action space, and c(s,a) denote the penalty function under the current action-state pair. The more actions an agent takes, the higher the reward, which incentivizes the agent to reduce the number of MCNS (Multiple Controlled Numbers).

[0130] To address the coverage overlap issue in HUD-IoT networks, the corresponding optimization problem is shown in Equation (20). Its objective is to minimize the number of MCNSs while ensuring conflict-free configuration. Within the CMDP framework, the number of MCNSs can be indirectly represented by the number of actions taken. Therefore, the cumulative reward can be used as an indicator of the number of MCNSs. Based on this, the optimization problem can be re-transformed into Equation (20):

[0131]

[0132] Among them, rewards(s t ,a t ) is r(s) t ,a t The full name of ) represents the state-action value at step t, cost(s t ,a t ) is c(s) t ,a t The full name is , which represents the state-action penalty value at step t, and maximize means maximizing the reward function value to find the policy.

[0133] like Figure 4 As shown in the diagram, this schematic illustrates the methodological structure for addressing the HUD-IoT coverage overlap problem. In this framework, the agent analyzes the input state and performs an action, namely selecting an access point to extend the HAS graph chain. Then, rewards and penalties are collected and stored in a replay buffer for training the WCSAC algorithm.

[0134] Furthermore, in the WCSAC algorithm, an entropy regularization mechanism can be used to balance action policies. Specifically, the entropy regularization mechanism is a key part of balancing the exploratory and random aspects of action policies, and this mechanism enhances the robustness of this method. The magnitude of entropy reflects the aggressiveness or conservatism of the agent's exploratory strategy. In WCSAC, the value function V... π (s) will be affected by the entropy term in entropy regularization, and the value function V in WCSAC will be affected by this. π (s) is represented as follows:

[0135]

[0136] in, Let r(s) represent the expectation of the above random trajectory. t ,at This is a simplified form of the reward function, where γ represents the discount factor, controlling the weighted reward, and α represents the entropy coefficient, used to adjust the entropy weight. Represents the entropy in the current state, π(·|s t ) represents the strategy, in the current state s t The probability distribution of actions under the given conditions.

[0137] In this embodiment, the value function is represented by the value function V. π (s) is used to stabilize the calculation of the Q target value, reduce variance, accelerate training, and also serves to make conservative estimates of future uncertainties, so that the strategy remains robust in the worst environment.

[0138] Furthermore, WCSAC can employ two safety evaluation networks to evaluate the safety of action policies, i.e., using two neural networks to estimate the safety evaluation function. Compared to SAC-Lagrangian (a safety reinforcement learning algorithm combining maximum entropy reinforcement learning and the Lagrange multiplier method), this structure enables agents to assess safety and risk based on observations.

[0139] In this embodiment, the two neural networks proposed by WCSAC are two security evaluation networks. and Used to match action risk C π (s,a) is a Gaussian distribution.

[0140] In formula (22), Corresponding to the mean network, Corresponding variance network.

[0141]

[0142] in, This represents the probability distribution of the state-action penalty function fitted by the two neural networks.

[0143] Among them, C π (s,a) will be iteratively updated using the Bellman operator by playing back multiple rounds of data in the empirical replay buffer. Through iterative updates, C π (s,a) gradually approximates the distribution of the safe value. and It is also updated using the Bellman operator, and the update expression can be represented as:

[0144]

[0145] Where c(s,a) represents the state-action penalty value, s′ represents the state space of all possible states for the next step, p(s′|s,a) represents the probability that the current state-action leads to a state transition to s′, A represents the action space, a′ represents a single action in the entire action space, and π(a′|s′) represents the action a generated in state s′. ′ strategy, This represents the soft value of evaluating the worst-case scenario for the state-action relationship. This represents the soft value of the current strategy under the worst-case scenario.

[0146] The neural network learns its parameters through gradient descent of the loss function to approximate the target function. and Let μ and η be used to represent and Neural networks, i.e. Therefore, the WCSAC algorithm uses the difference caused by the Bellman update (see equations (23)(24)) as the loss function:

[0147]

[0148] Among them, J C (μ) represents the gradient of the Q-network, J V (η) represents the gradient of the V network. Representing the action space and state space, ΔQ(s) t ,a t ,μ) represents the estimation error of the Q-network, ΔV(s) t ,a t ,η) represents the estimation error of the V network, trace(ΔV(s) t ,a t ,η)) represents the trace of the matrix of network error V.

[0149] In this embodiment, the Bellman operator update mainly updates the probabilities of various state-action indicators. Since the probabilities of state-action indicators in the newly generated samples deviate from the original probability distribution, this embodiment uses the loss function to fit and update the difference in probability distribution, so that the probability distribution of state-action indicators is closer to the real situation.

[0150] The state-action value function is evaluated and approximated using a dual-Q network, also known as a dual-safety evaluation network. These network structures are similar to safety evaluation networks (including...). )same, The state-value function representing the reward value is: The simplified form. For the loss function of the Action Policy Network (Actor), it is composed of... and Co-generated, for The simplified form. The loss function of the action policy network can be expressed as the formula:

[0151]

[0152] Among them, J π (θ) represents the loss function of the action policy network. s t Belongs to the set of all states a t ~π θ Indicates a t Belongs to π θ Actions in the strategy, π θ Let π represent the action selection policy, θ represent the parameters of the action policy π, and β represent the entropy weights (Lagrange multipliers) used to control the randomness of the action policy π and determine the relative importance of the entropy term with respect to reward and cost. θ (a t |s t () represents the action selection strategy in the current state. The state-value function represents the reward value. The state-value function represents the penalty value, and κ' represents the adaptive safety weights (Lagrange multipliers).

[0153] In this embodiment, the loss function serves to provide gradients to the neural network, enabling it to learn. Similarly, the loss function provided here is used to help the action policy network fit weights that better reflect reality.

[0154] This can be achieved by minimizing the loss function J. s (κ') is used to learn adaptive safety weights κ':

[0155]

[0156] Among them, J s (κ') represents the gradient of the safety weights, and d represents the safety threshold.

[0157] Therefore, when When the threshold is reached, κ' decreases; conversely, when the threshold is reached, κ' increases. This allows for the learning of adaptive safety weights κ' to further emphasize safety. The two soft Q functions are... These are used for the reward function and the safety evaluation network, respectively, and gradient descent training is performed using the minimum Q-value. After obtaining the loss function, the parameters are updated using a gradient-based optimization method. Here, the loss function refers to the loss function of the action policy network. and The loss function is defined by the parameters, which refer to the weights of the entire neural network.

[0158] In this embodiment, a soft update strategy can also be used to update the parameters of the target network, and its expression is as follows:

[0159] θ - ←η'θ+(1-η')θ - (28)

[0160] Where θ is the soft update parameter, the larger θ is, the faster the synchronization speed of the target network. η' represents the soft update coefficient, or the ratio of new to old loss, where θ can represent the parameter of the action policy π. The target network refers to the dual-Q network or the security evaluation network. and Their respective target networks, θ - This is the value after the soft update parameter is updated.

[0161] At this point, the algorithm is integrated with the CMDP model via WCSAC. The agent trained after integration explores partial order relationships and conflict values ​​for access point merging in the environment, storing this experience in a replay buffer. The experience in the replay buffer is used to iteratively update the reward function and the safety review network. During the exploration process, the agent's exploration level is controlled by entropy, which is particularly important in the early stages but should be gradually reduced over time.

[0162] Through training, the MCNS algorithm based on WCSAC was implemented, which solved the problem of communication resource reuse and coverage overlap conflict in HUD-IoT networks.

[0163] The following section simulates the MCNS algorithm based on WCSAC. The simulation environment is built on a general-purpose computer equipped with an Intel Xeon 6242R CPU (Central Processing Unit), 64GB of memory, and two NVIDIA RTX 3080Ti GPUs (Graphics Processing Units). The WCSAC model is implemented using PyTorch (an open-source deep learning framework based on Python), and the neural network is trained using an adaptive moment estimation optimizer (Adam). The simulation environment also supports the computation of partial order relations required to construct the proposed method.

[0164] During WCSAC training, the learning rate was set to 0.0001, and the β parameter ranged from 0.9 to 0.999. The environment parameter was set to 20. The simulated HUD-IoT area was a 1000m × 1000m square, with the number of APs and BSs ranging from 10 to 200, and their locations were randomly distributed.

[0165] The following is a brief description of the comparison algorithm used for evaluation.

[0166] Independent Set: This algorithm employs a greedy strategy, iteratively grouping nodes into independent sets and assigning colors to these sets. The implementation used in this simulation is based on the Python networkx library.

[0167] Random Sequential: This algorithm randomly sorts the nodes and then applies a greedy coloring strategy in that order. This implementation is also based on the networkx library.

[0168] Figure 5 and Figure 6 The simulation demonstrates the changes in cumulative reward and cumulative cost during training. In this simulation, the number of APs and BSs were both set to 15. As the number of training rounds increases, the cumulative reward curve steadily rises, indicating that the agent's performance in selecting MCNSs continuously improves. Meanwhile, the cumulative cost curve gradually decreases and eventually approaches zero, indicating that overlap conflicts are effectively avoided during training.

[0169] Figure 7 The performance comparison with other algorithms is shown. As the number of nodes increases, when the number of nodes reaches 160, the method proposed in this embodiment reduces the number of MCNS by approximately 6% compared to the random sequence algorithm and by approximately 9.6% compared to the independent set algorithm. This indicates that the proposed method is more efficient in communication resource allocation and can better support the connection needs of ultra-dense IoT devices.

[0170] Figure 8 The performance is further demonstrated under increased edge density, simulating a more severe conflict scenario. Under this condition, the method proposed in this embodiment still maintains a lower number of MCNS than the comparative algorithms, exhibiting superior performance stability. Meanwhile, WCSAC continuously reduces the number of MCNS, demonstrating better adaptability and efficiency in the HUD-IoT environment.

[0171] Furthermore, the present invention also provides an ultra-dense heterogeneous Internet of Things (IoT) resource allocation device based on undirected graphs and WCSAC, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can implement the above-mentioned ultra-dense heterogeneous IoT resource allocation method based on undirected graphs and WCSAC.

[0172] In summary, this invention provides a resource allocation method for ultra-dense heterogeneous IoT based on undirected graphs and WCSAC. The proposed algorithm employs a reinforcement learning framework, autonomously generating conflict-free solutions through self-learning. Unlike traditional methods that require pre-setting initial states, this method utilizes global observation of the solution space, eliminating the dependency on initialization. By modeling the task as a restricted Markov decision process, the algorithm naturally achieves no initialization operation. Furthermore, the agent's optimization trajectory is guided by a carefully designed reward function, which penalizes conflicts and prioritizes feasible solutions. Additionally, this invention's conflict-free resource reuse method for HUD-IoT scenarios can cover as many nodes as possible with the maximum conflict-free node set, effectively reducing the required resource amount while ensuring conflict-free resource reuse. This method effectively eliminates interference during resource reuse, achieving a resource consumption reduction of up to approximately 9% compared to traditional methods, significantly improving overall resource utilization efficiency.

[0173] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0174] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0175] Finally, it should be noted that the above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention should be considered equivalent substitutions and are included within the protection scope of the present invention.

Claims

1. A method for allocating ultra-dense heterogeneous IoT resources based on undirected graphs and WCSAC, characterized in that, include: Construct a communication network architecture for HUD-IoT scenarios, which includes multiple base stations, access points, and IoT devices; A conflict graph is established to represent the access point conflict situation in the communication network architecture, and a conflict-free graph is constructed based on the conflict graph; A partial order relation is established on the conflict-free graph of the communication network architecture, and a Hasse graph is constructed based on the partial order relation to represent the linear chain structure of the maximum conflict-free node set. The conflict-free resource reuse optimization problem of the aforementioned communication network architecture is established. The WCSAC reinforcement learning algorithm is used to train an agent under the constrained Markov decision process framework. The trained agent outputs an action policy based on the HASS diagram to obtain the final maximum set of conflict-free nodes, thereby realizing conflict-free resource reuse in the HUD-IoT scenario. The action policy is to select two access points or two maximum sets of conflict-free nodes for merging.

2. The method as described in claim 1, characterized in that, Determine the coverage area of ​​the base station and access point, and establish the conflict graph based on the coverage area and Euclidean distance. The adjacency matrix of the conflict graph satisfies: Among them, M adj(i,j) D represents the adjacency matrix of the conflict graph. i,j Let l represent the Euclidean distance between the i-th and j-th access points. i l j Let represent the coverage radii of the i-th and j-th access points, respectively.

3. The method as described in claim 1, characterized in that, Based on the conflict graph, access points that can share the channel without interference are determined to construct a conflict-free graph and obtain the adjacency matrix of the conflict-free graph.

4. The method as described in claim 1, characterized in that, Establishing a partial order relation includes: For any set of access points A, let the partial order relation be... If for all If y, z∈A, and the partial order relation R satisfies antisymmetry and transitivity, then the partial order relation R is determined to be a partial order relation on the set of access points A, and the ordered pairs (R, A) constitute a partial order set, where x, y, z are the first to third access points, respectively.

5. The method as described in claim 1, characterized in that, Constructing a Hasse diagram includes: Node pairs that satisfy transitivity and antisymmetry are selected based on partial order relations; The node pairs are filtered to select the edges that need to be retained and to remove redundant edges. The linear chain structure is formed based on the retained node pairs and edges; Multiple linear chain structures are merged to construct the Hasse diagram.

6. The method as described in claim 1, characterized in that, The conflict-free resource reuse optimization problem is to identify and output the maximum conflict-free node set to maximize channel reuse efficiency while minimizing the maximum number of conflict-free nodes.

7. The method as described in claim 1, characterized in that, The WCSAC reinforcement learning algorithm is used to train the agent within the framework of constrained Markov decision processes, including: The state space is defined as the partial order relationship between nodes, and the action space is defined as the node merging operation. The agent is trained, and an entropy regularization mechanism is used to balance the action policy. Two safety evaluation networks are used to evaluate the safety of the action policy. The two safety evaluation networks are the mean network and the variance network, respectively. The two safety evaluation networks are updated by the Bellman operator.

8. The method as described in claim 7, characterized in that, When training the agent, a reward function is used to incentivize the agent to reduce the number of the maximum number of conflict-free nodes based on a non-linear incentive mechanism; a cost function is used to apply negative feedback to merging actions that cause conflicts to ensure interference-free communication.

9. The method according to any one of claims 1 to 8, characterized in that, The final output set of maximum conflict-free nodes satisfies the following conditions: it covers all access points; there are no conflicts between access points within each maximum conflict-free node set; and there are no channel multiplexing conflicts between different maximum conflict-free node sets.

10. A resource allocation device for ultra-dense heterogeneous Internet of Things based on undirected graphs and WCSAC, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the method as described in any one of claims 1 to 9.

Images