An end-edge collaborative access control and resource allocation method for a multi-loop wireless networked control system

Abstract

The application provides an end-edge collaborative access control and resource allocation method for a multi-loop wireless networked control system, which comprises the following steps: constructing an end-edge collaborative control-communication architecture of the multi-loop wireless networked control system; modeling an end-edge collaborative access control and resource allocation problem in the end-edge collaborative control-communication architecture into a hierarchical semi-Markov decision model; designing an end-side shared decision network and an edge-side tokenization decision network, and performing end-edge collaborative reinforcement learning training based on the hierarchical semi-Markov decision model; determining whether to initiate an edge-assisted control request and the corresponding action duration time slot according to the end-side local state of the wireless control loop by the trained end-side shared decision network; performing access authorization according to a dynamic request set by the trained edge-side tokenization decision network, and generating a bandwidth and transmit power allocation result meeting the wireless resource constraint. The application can reduce high-frequency request interaction, improve resource coordination capability in a dynamic request scenario, reduce communication energy consumption, and improve control performance.

Description

Technical Field

[0001] This invention relates to the technical field of access control and wireless resource allocation, and in particular to an end-edge collaborative access control and resource allocation method for multi-loop wireless network control systems. Background Technology

[0002] With the development of the Industrial Internet, intelligent manufacturing, and wireless control technologies, an increasing number of industrial devices are accessing control systems via wireless networks to achieve status awareness, control command transmission, and collaborative operation. Wireless networked control systems can reduce cable deployment costs, improve system flexibility, and are suitable for industrial scenarios such as mobile robots, automated guided vehicles, robotic arms, and production line equipment. In multi-loop wireless networked control systems, multiple control loops communicate with an edge server by sharing wireless resources. The edge server can generate edge-assisted control inputs based on uploaded status information to enhance control performance and system collaboration capabilities.

[0003] However, wireless resources such as subcarriers, bandwidth, and transmit power are typically limited in industrial wireless networks. When multiple control loops simultaneously request edge-assisted control, access contention and resource conflicts can easily arise. If a control loop fails to obtain effective edge services in a timely manner, it may lead to status upload failure, control input delay, or increased control error, thereby causing a decline in control performance or even exceeding the error threshold. Therefore, how to perform access control and wireless resource allocation based on the control status, wireless link status, and resource occupancy of each control loop under limited wireless resource conditions is a key issue in multi-loop wireless networked control systems.

[0004] Existing access control and resource allocation methods mainly include centralized scheduling methods and edge-distributed methods. Centralized scheduling methods typically require edge servers to continuously collect global state information from all control loops and perform unified optimization in each time slot, which easily leads to significant communication overhead and computational complexity, and lacks scalability when the number of control loops changes. While edge-distributed methods can reduce the collection of global information, they struggle to utilize global resource contention information to complete access authorization and radio resource coordination.

[0005] In recent years, deep reinforcement learning methods have attracted widespread attention. Their ability to learn adaptive policies through interaction with the environment has been used to solve wireless access control and resource allocation problems. However, existing methods are mostly based on fixed-slot Markov decision processes, where the decision agent typically needs to repeatedly update request decisions in each slot, easily leading to unnecessary high-frequency request interactions and policy fluctuations. Furthermore, existing methods often use fixed-dimensional state inputs, making it difficult to adapt to scenarios with dynamically changing edge request sets and to adequately characterize the resource competition relationships between multiple control loops. Therefore, there is an urgent need for an edge-end collaborative access control and resource allocation method for multi-loop wireless networked control systems, which can balance control performance, communication energy consumption, and decision overhead under conditions of limited wireless resources and time-varying industrial channels. Summary of the Invention

[0006] This invention aims to address the challenge of balancing closed-loop control performance, communication energy consumption, and end-edge interaction overhead in multi-loop wireless networked control systems under conditions of limited wireless resources and time-varying industrial wireless channels. To this end, this invention proposes an end-edge collaborative access control and resource allocation method for multi-loop wireless networked control systems. This method reduces redundant decisions and requests through joint decision-making based on end-side requests and the number of persistent time slots, and achieves access authorization and joint allocation of wireless resources for dynamic request sets through dynamic tokenization decision-making at the edge side.

[0007] To achieve the above objectives, the technical solution of the present invention is as follows: an end-edge cooperative access control and resource allocation method for a multi-loop wireless network control system, comprising the following steps:

[0008] S1: Construct an end-edge collaborative control-communication architecture for a multi-loop wireless networked control system;

[0009] S2: Model the end-side collaborative access control and resource allocation problem in the end-side collaborative control-communication architecture as a hierarchical semi-Markov decision model;

[0010] S3: Design a shared decision network on the end side and a tokenized decision network on the edge side, and perform end-edge collaborative reinforcement learning training based on a hierarchical semi-Markov decision model; the trained shared decision network on the end side determines whether to initiate an edge-assisted control request and the corresponding action duration slot number based on the local state of the end side of the wireless control loop; the trained tokenized decision network on the edge side executes access authorization based on the dynamic request set and generates bandwidth and transmit power allocation results that satisfy wireless resource constraints.

[0011] Preferably, the end-edge collaborative control-communication architecture includes There are one wireless control loop and one edge server. The set of wireless control loops is denoted as . Divide the runtime into A time slot of equal length, time slot index ; in each time slot Within the system, each wireless control loop determines whether to request edge-assisted control based on its own local control state and wireless communication state, thereby determining whether an edge-assisted control request is generated. The edge server performs access authorization selection, bandwidth allocation, and transmit power allocation based on the received edge-assisted control request, the current wireless resource state, and the state of the control loop being served.

[0012] Preferably, the hierarchical semi-Markov decision model includes an end-side semi-Markov decision model and an edge-side semi-Markov decision model. The end-side semi-Markov decision model is used to describe how each wireless control loop determines whether to request edge-assisted control based on the end-side local state, and the number of time slots that need to be maintained for the edge-assisted control request decision.

[0013] The edge-side semi-Markov decision model is used to describe how the edge server performs centralized resource coordination based on dynamically arriving edge-assisted control requests and the current state of wireless resources; the edge-side semi-Markov decision model performs authorization selection and resource allocation based on the set of wireless control loops currently participating in resource contention;

[0014] The interaction coupling mechanism between the end-side semi-Markov decision model and the edge-side semi-Markov decision model:

[0015] When the wireless control loop arrives at the edge-side decision time, it determines the edge-side decision action based on the current edge-side local state. When the edge-side decision action indicates initiating an edge-assisted control request, the corresponding wireless control loop is added to the new arrival request set. When the edge server receives a new edge-assisted control request, it constructs the edge-side dynamic decision state based on the new arrival request, the in-service control loop, and available wireless resources, and performs new access authorization and wireless resource allocation. The results of the new access authorization and wireless resource allocation determine whether the state upload can be completed within the specified time delay, and further determine whether effective edge control is formed. The effective edge control result changes the control input adopted by the controlled object, causing changes in the state and control error of the controlled object. After the sensor re-acquires the state, it updates the edge-side local state and the edge-side dynamic decision state.

[0016] Preferably, the edge-side decision-making action affects the selection of new access authorization and the allocation of radio resources on the edge side by changing the edge-assisted control request set; the selection of new access authorization and the allocation of radio resources on the edge side further affect the subsequent edge-side local state and edge-side decision-making action by changing the state upload result, the effective edge control result and the state of the controlled object;

[0017] No. A wireless control loop at the end-side decision time The resulting end-side semi-Markov transfer samples are:

[0018] ;

[0019] in, Indicates the first The wireless control loop in the first Local state of the endpoint at each endpoint decision moment Indicates the next decision-making moment. The local state of the end side, For edge-side decision-making actions, This indicates the number of time slots during which the decision-making action is performed on the endpoint. For the first The first wireless control loop Individual decision-making moment Range of returns;

[0020] No. The edge-side semi-Markov transition samples generated at each edge-side decision time are:

[0021] ;

[0022] in, Indicates the first The dynamic decision-making state of the edge side at each edge-side decision-making moment. Indicates the first The dynamic decision-making state of the edge side at each edge-side decision-making moment; For the first Actions executed at the edge of the decision-making process; For the first Edge-side interval return at each edge-side decision moment; This represents the number of time slots between two adjacent edge-side decision moments.

[0023] Preferably, the end-side local state of the end-side semi-Markov decision model includes local channel state information, waiting time, security margin, and security margin change trend; The local state at the end of each wireless control loop is represented as follows:

[0024] ;

[0025] in, Indicates the first A wireless control loop in the time slot The obtained local channel state information, For the first A wireless control loop in the time slot Waiting time, For the first A wireless control loop in the time slot Safety margin, For safety margin Trends;

[0026] No. Each wireless control loop is based on the local state of the end side. The output edge-side decision action is a joint action of request and duration slot number: ;in, Indicates the first The wireless control loop in the first Individual decision-making moment An indicator variable for whether to initiate an edge-assisted control request. Representation and indicator variables The corresponding number of time slots for edge-side decision-making actions. Maximum duration;

[0027] No. The first wireless control loop Individual decision-making moment The range of returns is:

[0028] ;

[0029] in, This represents the decision cost incurred by the endpoint in executing a decision once at a single decision moment. The discount factor; the first A wireless control loop in the time slot End-side instant feedback

[0030] ;

[0031] in, Indicates the first A wireless control loop in the time slot Tracking error Relative to threshold Safety margin, The energy consumed to initiate an edge-assisted control request. For the first A wireless control loop in the time slot Edge-assisted control request variables; Indicates the first A wireless control loop in the time slot Whether edge-assisted control is effectively implemented. For the first A wireless control loop in the time slot Effective edge control indicator variables, Indicates tracking error Exceeding the threshold Indicator functions, , , , All are positive weighting coefficients;

[0032] The edge-side dynamic decision state is determined by the first... Each edge decision moment The current set of wireless control loops involved in authorization determination and resource preservation The status of each wireless control loop and the current status of available wireless resources constitute the basis for... ;in, Indicates the first The status of each wireless control loop For the first A wireless control loop at the edge-side decision-making time Are there active edge-assisted control request variables? Indicates the first The current service status of each wireless control loop. Indicates the first A wireless control loop at the edge-side decision-making time The number of remaining service slots; Indicates the decision-making moment at the edge. The number of remaining subcarriers that can be used to add new access licenses;

[0033] No. Actions executed at the edge of the decision-making process. ;in, and They represent the first time. The edge decision time is assigned to the first Bandwidth and transmit power of each wireless control loop; For edge servers in the first Each edge decision moment The newly arrived edge-assisted control requests form a new arrival request set. Select the new access authorization set;

[0034] No. Each edge decision moment The edge-side interval return is: ;in, 1 indicates the first Slot offset index within each edge-side decision moment; edge-side instant report ;in, Indicates the first A wireless control loop in the time slot The transmission power, Indicates the maximum permissible transmission power; for Uplink latency of each wireless control loop Indicates the first A wireless control loop in the time slot The scheduling variable; This indicates a penalty for edge service failure. , , All are positive weighting coefficients.

[0035] Preferably, the end-side shared decision network is deployed in each wireless control loop. The end-side shared decision network of each wireless control loop uses the same network structure and network parameters, and independently generates end-side decision actions based on their respective end-side local states. The edge-side tokenized decision network is deployed on the edge server to process resource coordination sets with dynamically changing scales and output access authorization and wireless resource allocation results. The end-side shared decision network and the edge-side tokenized decision network interact through edge-assisted control requests, access authorization results, status upload results, and valid edge control indicator variables.

[0036] The edge-side tokenization decision network constructs a dynamic token set for the edge side based on the wireless control loops currently participating in resource coordination;

[0037] Training end-side shared decision networks and edge-side tokenized decision networks using semi-Markov transfer samples from the end and edge sides.

[0038] Preferably, the end-side shared decision network uses the end-side local state As input, the candidate actions on the endpoint are determined by the indicator variable. and Combining and consisting of 2 Each candidate edge-side decision action is assigned a value corresponding to a specific action in the shared decision network output. The candidate edge-side decision action with the highest value is then selected.

[0039] In the Each edge decision moment According to the wireless control loop that participates in authorization judgment and resource maintenance Construct a dynamic token set for the edge side: for any Construction and the first The edge-side token corresponding to each wireless control loop:

[0040] ;

[0041] in, Indicates the first A wireless control loop at the edge-side decision-making time The corresponding end-side local state, Indicates the first An edge control request indicator variable for a wireless control loop Indicates the first Is each wireless control loop in edge service mode? Indicates the first The number of remaining service slots for each wireless control loop. This indicates the decision-making time of the edge server at the edge. The number of remaining available subcarriers;

[0042] The edge-side dynamic token set is represented as: ;

[0043] The end-side shared decision network and the edge-side tokenized decision network update parameters based on the end-side semi-Markov transition samples and the edge-side semi-Markov transition samples, respectively. End-side decision actions affect the edge-side decision state by changing the edge-assisted control request set. Edge-side access authorization and wireless resource allocation results affect subsequent end-side decisions by changing communication results and the state of the controlled object. End-side collaborative training is achieved in the common interaction process.

[0044] During the training phase, the parameters of the end-side shared decision network, the parameters of the edge-side tokenized decision network, the end-side experience cache, and the edge-side trajectory cache are maintained. During the training interaction, each wireless control loop generates end-side decision actions based on its own end-side local state, and the edge server generates access authorization results, bandwidth allocation results, and transmit power allocation results based on the edge-side dynamic token set. The state of the controlled object and the wireless communication state are advanced according to the end-side decision actions and the edge-side decision actions, and the end-side interval reward and the edge-side interval reward are calculated respectively. When the end-side decision action is completed, the corresponding end-side semi-Markov transition sample is generated and stored in the end-side experience cache. When the next edge-side decision time is reached, the corresponding edge-side semi-Markov transition sample is generated and stored in the edge-side trajectory cache.

[0045] Preferably, the edge-side shared decision network adopts a dual-depth Q-network based on a competitive structure, including an evaluation Q-network and a target Q-network; the evaluation Q-network is used to output the current action value of each candidate edge-side decision action based on the current edge-side local state, and the target Q-network is used to calculate the target action value during the training phase; the competitive structure decomposes the action value function into state values. and action advantages The current action value of each candidate endpoint decision action is:

[0046] ;

[0047] in, Represents the joint set of candidate decision actions on the endpoint. Number of actions included Represents the set of action unions The action index used during traversal; This represents the trainable parameters for evaluating the Q-network; This indicates the evaluation of the Q-network's estimation of the action value of state-action pairs;

[0048] The edge-side tokenization decision network adopts an actor-commentator structure, including an actor network and a commentator network. The actor network includes an embedding module, an encoding module, an authorization selection module, and a resource allocation module, used to generate access authorization results and radio resource allocation results. The commentator network is used to estimate the state value of the edge-side dynamic decision state based on the edge-side dynamic token set during the training phase. The embedding module of the actor network normalizes the continuous features in the edge-side tokens, and then maps the edge-side tokens to initial hidden features of a unified dimension through at least one fully connected layer. The encoding module includes at least one Transformer encoding block, each Transformer encoding block including a multi-head attention sublayer and a feedforward neural network sublayer. Both the multi-head attention sublayer and the feedforward neural network sublayer employ layer normalization and residual connections. Through multi-head self-attention computation of the multi-head attention sublayer, the hidden features of each radio control loop adaptively fuse with the current resource-coordinated radio control loop set. The request status, control status, and resource occupancy status of other wireless control loops are analyzed to obtain enhanced relationship features characterizing the resource contention relationship between loops; the authorization selection module only considers newly arriving request sets. The radio control loop in the system performs a new license determination and outputs a license priority score based on the relationship enhancement features. It determines the number of new licenses based on the number of new arrival requests and the number of remaining available subcarriers. Finally, based on the license priority score and the number of new access licenses, it obtains a set of new access licenses. ; Add new access authorization sets With the set of wireless control loops currently in service By merging, we obtain the set of wireless control loops currently participating in wireless resource allocation. The resource allocation module determines the set of wireless control loops currently participating in wireless resource allocation. The relationship between each wireless control loop is enhanced, and the allocation parameters corresponding to the output bandwidth allocation random variable and transmit power allocation random variable are obtained. After normalization and resource constraint mapping, the actual bandwidth and transmit power allocation results are obtained.

[0049] Preferably, during the training phase, dual Q-learning is used to calculate the target action value. End-side semi-Markov transition samples are sampled from the end-side experience cache. The evaluation Q-network is updated based on the target action value, and the target Q-network is synchronized with the evaluation Q-network parameters. The evaluation Q-network selects the optimal candidate end-side decision action corresponding to the next end-side local state. The value of a semi-Markov objective action is calculated using a target Q-network: ;in, The network parameters of the target Q-network;

[0050] The set of relationship-enhanced features encoded by the Transformer coding block is as follows:

[0051] ;

[0052] in, This represents the encoding function jointly constituted by the embedding module and the encoding module. Indicates the first The set of dynamic tokens on the edge side at each edge-side decision moment. Indicates the first Enhanced relationship features of a wireless control loop after fusing information from other participating resource coordination loops;

[0053] Authorization priority score: ;

[0054] The number of new authorizations is: ;

[0055] The newly added access authorization set is: ;

[0056] in, This represents a mapping function that outputs an authorization priority score based on relation enhancement features; This indicates that the request set arrives again in descending order of authorization priority score. Before the election Operation of a wireless control loop.

[0057] For the edge-side tokenized decision network, the edge-side temporal difference residual is calculated based on the edge-side semi-Markov transition samples:

[0058] ;

[0059] in, This represents the state value of the commentator's network output. This represents the parameters of the critic network. This is a discount term; the edge-side dominance estimate is calculated based on the edge-side temporal difference residual, and the edge-side tokenized decision network is updated based on the edge-side dominance estimate.

[0060] Preferably, the edge server determines the order based on the newly arrived request set. and the set of wireless control loops currently in service Construct the set of wireless control loops currently involved in authorization determination and resource preservation: ;

[0061] Edge servers in the Each edge decision moment The set of wireless control loops corresponding to the newly arrived edge-assisted control requests. Select Add Access Authorization Set It also allocates bandwidth and transmit power to newly granted access and wireless control loops that are receiving edge-assisted control.

[0062] The newly added access authorization set satisfies: ;in, Indicates the newly added access authorization set The number of wireless control loops in the middle;

[0063] At least one fully connected layer of the end-side shared decision network is a noisy linear layer, and the weights and biases of the noisy linear layer are expressed as follows: and ;in, , , and All are learnable parameters. and The noise is random; during training, random noise is resampled at a preset frequency so that the edge-side shared decision network generates different action value perturbations in different edge-side local states, thereby realizing state-related action exploration; during the deterministic execution phase, learnable parameters are used to output action values.

[0064] Repeat the edge-to-edge interactive sampling and network update process until the preset maximum number of training rounds is reached. After training is completed, the evaluation Q network parameter copies in the edge-side shared decision network are deployed to each wireless control loop, and the actor network parameters of the edge-side tokenized decision network are deployed to the edge server.

[0065] During operation, each wireless control loop determines whether to initiate an edge-assisted control request and the corresponding duration of action slots based on its own local end-side state. The edge server constructs an edge-side dynamic token set based on the set of newly arrived requests and the set of wireless control loops currently being served, performs deterministic Top-K access authorization, and generates resource allocation results that satisfy system bandwidth and maximum transmit power constraints.

[0066] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0067] 1. This invention models the end-edge collaborative access control and resource allocation problem as a hierarchical semi-Markov decision model, enabling end-side request decisions and edge-side resource coordination decisions to be made at different decision times, thus better conforming to the actual characteristics of continuous evolution of control state, asynchronous arrival of access requests, and dynamic occupation of edge resources in multi-loop wireless network control systems;

[0068] 2. The present invention designs a request-persistent time slot number joint action on the end side, so that the wireless control loop not only determines whether to request edge-assisted control, but also determines the holding time of the request decision, which can reduce unnecessary repeated requests and high-frequency end-side interaction, and reduce decision overhead and communication overhead.

[0069] 3. This invention constructs a dynamic token set at the edge and extracts the resource contention characteristics between each wireless control loop through a tokenization policy network, so that the edge server only processes the wireless control loops currently participating in resource contention and resource preservation, thereby improving the adaptability of the edge-side policy to different system scales and the number of dynamic requests.

[0070] 4. This invention performs Top-K licensing selection based on resource contention characteristics and jointly allocates radio resources for new licensed control loops and in-service control loops. It can coordinate the edge-assisted control requirements of multiple radio control loops under constraints of limited subcarriers, total system bandwidth, and transmit power, thereby improving service reliability and resource utilization efficiency under limited radio resource conditions. Attached Figure Description

[0071] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0072] Figure 1 This is a diagram of the edge-end collaboration architecture provided in an embodiment of the present invention.

[0073] Figure 2 This is a schematic diagram of the workflow of the multi-loop wireless network control system provided in an embodiment of the present invention.

[0074] Figure 3 A schematic diagram of the access control and resource allocation algorithm model based on edge-end collaborative reinforcement learning provided for the instantiation of this invention.

[0075] Figure 4This is a flowchart of an end-edge collaborative access control and resource allocation method for a multi-loop wireless network control system provided in an embodiment of the present invention. Detailed Implementation

[0076] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0077] like Figure 4 As shown, an end-edge collaborative access control and resource allocation method for a multi-loop wireless networked control system includes establishing an end-edge collaborative control-communication architecture for the multi-loop wireless networked control system; modeling the access control and resource allocation problem as a hierarchical semi-Markov decision model, enabling the end side to generate access requests and action duration slots, and the edge side to perform authorization selection and wireless resource allocation based on a dynamic request set; designing an end-side shared decision network and an edge-side tokenized decision network, and performing end-edge collaborative reinforcement learning training based on semi-Markov transition samples. The edge side extracts resource contention characteristics between control loops by constructing a dynamic token set, and performs authorization selection and joint allocation of bandwidth and transmit power. This invention can reduce high-frequency request interactions, improve resource coordination capabilities in dynamic request scenarios, reduce communication energy consumption, and improve control performance. The implementation steps of this invention are as follows:

[0078] S1: Constructing an end-edge collaborative control-communication architecture for a multi-loop wireless networked control system.

[0079] like Figure 1 and Figure 2 As shown, this embodiment first establishes the end-edge collaborative control-communication architecture of the multi-loop wireless network control system, including... There are one wireless control loop and one edge server. The set of wireless control loops is denoted as . Each wireless control loop includes a controlled object, sensors, a local controller, and actuators. The controlled object is the physical device or process receiving control; sensors are connected to the controlled object to collect its status information; and actuators are connected to the controlled object to apply control actions based on received control inputs. The status information collected by the sensors is provided to the local controller and uploaded to the edge server via a wireless link when edge-assisted control is requested. Local control inputs generated by the local controller or edge control inputs generated by the edge server are provided to the actuators, thus forming a closed-loop control system. The local controller generates local control inputs when effective edge-assisted control is not available, while the edge server generates edge control inputs after the control loop has acquired wireless resources and completed status uploads.

[0080] System uptime is divided into There are several equal-length time slots, with the time slot index being... In each time slot Within the system, each wireless control loop determines whether to request edge-assisted control based on its own local control state and wireless communication state, thereby determining whether an edge-assisted control request is generated. The edge server performs access authorization selection, bandwidth allocation, and transmit power allocation based on the received edge-assisted control request, the current wireless resource state, and the state of the control loop being served.

[0081] For the A wireless control loop, whose controlled object is modeled as a discrete-time linear time-invariant system:

[0082] ;

[0083] ;

[0084] in, For system status, To control the input, For system output, , and These are the state transition matrix, input matrix, and output matrix, respectively.

[0085] set up For the first A wireless control loop in the time slot The effective edge control indicator variable. When When, it indicates the first Each wireless control loop obtains effective edge-assisted control in the current time slot; when When, it indicates the first The wireless control loop uses end-side local control. Then the... The control input for each wireless control loop is represented as follows:

[0086] ;

[0087] in, This represents the local control input generated by the local controller. This represents the edge control input generated by the edge server.

[0088] To characterize the operating state of the control loop, let... For the first A wireless control loop in the time slot If the expected output is obtained, then the tracking error is expressed as:

[0089] ;

[0090] Let the error threshold be ,when At that time, the first The tracking error of the wireless control loop exceeds the error threshold. Further, the first... The safety margin for each wireless control loop is:

[0091] ;

[0092] The trend of safety margin change is defined as follows:

[0093] ;

[0094] Among them, the smaller the safety margin, the closer the control loop is to the tolerance boundary; when the safety margin changes in a negative direction, it indicates that the control performance is deteriorating.

[0095] On the communication side, the wireless control loop communicates with the edge server via a wireless link and uses orthogonal frequency division multiple access for uplink access. Assume the system has... One available subcarrier, and Within the same time slot, each scheduled radio control loop occupies at most one subcarrier; therefore, the number of simultaneously scheduled radio control loops does not exceed [a certain limit]. .

[0096] In the time slot In the middle, set For the first An edge-assisted control request variable for a wireless control loop Indicates the first One wireless control loop initiates an edge-assisted control request. Indicates the first Each wireless control loop does not initiate an edge-assisted control request. Assume... For edge servers to the first Authorized variables for each wireless control loop, Indicates the edge server authorization number One wireless control loop is connected. This indicates that the edge server has not authorized the first... One wireless control loop is connected. The scheduling variables are jointly determined by the edge-assisted control request variable and the authorization variable:

[0097] ;

[0098] in, Indicates the first A wireless control loop in the time slot The status is uploaded by the scheduled entity. Indicates the first A wireless control loop in the time slot Not scheduled.

[0099] For the scheduled wireless control loop, the edge server further allocates bandwidth and transmit power to it. Let... Indicates assignment to the first The bandwidth of a wireless control loop Represents the total system bandwidth; let... Indicates the first The transmit power of each wireless control loop This indicates the maximum permissible transmission power.

[0100] Under the communication conditions of industrial wireless links, let's assume... Indicates the first A wireless control loop in the time slot The effective uplink channel coefficient, whose value is determined by the time slot. The wireless propagation environment within the area has been determined. For the noise power spectral density, then when the... When a wireless control loop is scheduled, its uplink transmission rate is expressed as:

[0101] ;

[0102] Let the amount of data uploaded each time be... Bit, then the first The uplink delay of each wireless control loop is:

[0103] ;

[0104] Let the maximum allowed uplink transmission delay be Then the first The transmission success indicator variable for each wireless control loop is defined as follows:

[0105] ;

[0106] Therefore, the effective edge control indicator variable is expressed as: That is, only when the first The first wireless control loop is considered complete only when it initiates an edge-assisted control request, obtains authorization from the edge server, and completes state upload within the time delay constraint. The first wireless control loop achieves effective edge-assisted control; otherwise, the second... Each wireless control loop continues to use end-side local control input.

[0107] Based on the aforementioned edge-edge collaborative control-communication architecture, the wireless control loop on the edge can determine whether to request edge-assisted control based on its own local control and communication states. The edge server can centrally authorize and allocate resources based on the request states of multiple wireless control loops and the wireless resource states. This provides a foundation for subsequently constructing a hierarchical semi-Markov decision model and designing an edge-edge collaborative reinforcement learning algorithm.

[0108] S2: Model the end-edge collaborative access control and resource allocation problem as a hierarchical semi-Markov decision model.

[0109] Because the end-side control loops and edge servers in a multi-loop wireless networked control system have different observation information and decision-making responsibilities, if only a single centralized decision-making model is used, the edge server needs to continuously collect the status information of all wireless control loops, which easily increases communication overhead and computational burden. If only an end-side distributed decision-making model is used, each control loop will find it difficult to obtain global wireless resource contention information, making it impossible to efficiently complete authorization selection and wireless resource allocation. Therefore, this embodiment uses a hierarchical semi-Markov decision model to describe end-side request decisions and edge-side resource coordination decisions. The hierarchical semi-Markov decision model includes an end-side semi-Markov decision model and an edge-side semi-Markov decision model.

[0110] S21: Establish a semi-Markov decision-making model on the end side;

[0111] The edge-side semi-Markov decision model describes how each control loop determines whether to request edge-assisted control based on its local state, and the number of time slots required to make the edge-assisted control request decision. The local state of the edge-side semi-Markov decision model includes local channel state information, waiting time, safety margin, and safety margin change trend. The local channel state information reflects the communication conditions between the wireless control loop and the edge server; the waiting time characterizes the number of time slots the wireless control loop has experienced since its last effective edge-assisted control; the safety margin characterizes the degree to which the wireless control loop has approached the error threshold boundary; and the safety margin change trend characterizes the direction of change in control performance.

[0112] No. The local state at the end of each wireless control loop is represented as follows:

[0113] ;

[0114] in, Indicates the first A wireless control loop in the time slot Available local channel state information, Waiting time is the number of time slots elapsed since the radio control loop last obtained valid edge service. For safety margin, This represents the trend of safety margin changes. Local state at the end. It also reflects communication conditions and the urgency of control, enabling the wireless control loop on the end side to generate edge-assisted control requests based on its actual service needs.

[0115] Let the first The first wireless control loop The decision-making time at each endpoint is At that decision-making moment The wireless control loop on the end side is based on the local state of the end side. The output end-side action is a combined request-duration slot number action:

[0116] ;

[0117] in, Indicates the first The wireless control loop in the first An indicator variable indicating whether to initiate an edge-assisted control request at each edge-side decision point. Representation and indicator variables The corresponding number of time slots for edge-side decision-making actions. The maximum duration. When the indicator variable At that time, the first A wireless control loop is continuously Maintain the edge-assisted control request state within each time slot; when the indicator variable At that time, the first A wireless control loop is continuously No edge-assisted control is actively requested within each time slot. After the duration ends, the radio control loop re-determines the request-duration time slot number. The next end-side decision time for this radio control loop is:

[0118] ;

[0119] Therefore, the requested duration slot number is not a pre-fixed system parameter, but is determined by the end-side control loop based on its own state. This is achieved through the combined action of the request and duration slot number. The wireless control loop not only determines whether to request edge-assisted control, but also determines the number of time slots that keep the request decision valid, thereby reducing unnecessary high-frequency request interactions. The end-side wireless control loop can reduce unnecessary repeated requests within adjacent time slots, and promptly re-make request decisions when the control state deteriorates.

[0120] Real-time feedback from the endpoint is used to evaluate the effectiveness of endpoint decision-making actions within the current time slot. Effectiveness includes control performance, request energy consumption, invalid requests, and the impact of errors exceeding thresholds. For the first... The real-time feedback from the end side of each wireless control loop is as follows:

[0121] ;

[0122] in, Indicates the error of the wireless control loop Relative to threshold Safety margin, Indicates the first A wireless control loop in the time slot The energy consumption of initiating an edge-assisted control request, of which Energy consumed to initiate an edge-assisted control request. Indicates the first A wireless control loop in the time slot Whether the initiated edge-assisted control is effectively controlled, i.e. . Indicates error Exceeding the threshold Indicator functions, , , , All are positive weighting coefficients.

[0123] For the One wireless control loop, interval report:

[0124] ;

[0125] in, This represents the decision overhead incurred by the endpoint in executing a decision at a single decision time. This interval return is used to evaluate the cumulative impact of the selected request-duration slot number joint action over its duration, enabling endpoint decision-making to simultaneously consider control safety margins, communication request energy consumption, and decision frequency. This is the discount factor.

[0126] Therefore, the first A wireless control loop at the end-side decision time The resulting end-side semi-Markov transfer samples are represented as follows:

[0127] ;

[0128] in, Indicates the first The wireless control loop in the first Local state of the endpoint at each endpoint decision moment This represents the local state of the endpoint at the next endpoint decision time.

[0129] S22: Establish a semi-Markov decision model for the edge side.

[0130] The edge-side semi-Markov decision model describes how the edge server performs centralized resource coordination based on dynamically arriving edge-assisted control requests and the current state of radio resources. Unlike end-side decision-making, edge-side decision-making involves resource contention among multiple radio control loops; therefore, the edge side needs to select licenses and allocate resources based on the set of control loops currently participating in resource contention.

[0131] Let the first Each edge decision moment is When the edge server receives a new edge-assisted control request, it determines the newly arrived request set. Meanwhile, since some wireless control loops may still be in edge service, the set of wireless control loops currently in service is defined as follows: The edge server constructs the current resource coordination set based on the set of newly arriving requests and the set of wireless control loops currently being served:

[0132] ;

[0133] in, This represents the set of wireless control loops currently involved in authorization determination and resource preservation.

[0134] Edge-side actions include license selection, bandwidth allocation, and transmit power allocation. Specifically, the edge server... Each edge decision moment The set of wireless control loops corresponding to the newly arrived edge-assisted control requests. Select Add Access Authorization Set It also allocates bandwidth and transmit power to newly granted access licenses and wireless control loops currently receiving edge-assisted control. The newly added access license set satisfies the following:

[0135] ;

[0136] in, Indicates the newly added access authorization set The number of wireless control loops in the middle, Indicates the decision-making moment at the edge. The number of remaining subcarriers that can be used to add new access licenses.

[0137] After determining the new license set, the edge server constructs the set of radio control loops currently participating in radio resource allocation:

[0138] ;

[0139] Among them, the wireless control loop set This includes newly licensed radio control loops and radio control loops currently in service. The status of each scheduled loop is:

[0140] ;

[0141] in, Indicates the first The status of a wireless control loop. Instruction No. A wireless control loop at the edge-side decision-making time Are there any active edge-assisted control requests? Indicates the first The current service status of each wireless control loop. Indicates the first The number of remaining service slots for each wireless control loop.

[0142] No. The actions executed at each edge decision point are as follows:

[0143] ;

[0144] in, Indicates the first The edge decision time is assigned to the first The bandwidth of a wireless control loop Indicates the first The edge decision time is assigned to the first The transmit power of each wireless control loop.

[0145] Edge-side real-time reports are used to evaluate the impact of edge-side authorization and resource allocation actions on communication energy consumption, edge service reliability, and control performance. Edge-side real-time reports are expressed as follows:

[0146] ;

[0147] in, Indicates the first The energy consumption of state upload transmission generated by each wireless control loop when obtaining edge services. The penalty for edge service failure is represented as: , An indicator function representing an error exceeding a threshold. , , All are positive weighting coefficients. Therefore, in the th... At each edge-side decision time, the edge-side interval return is:

[0148] ;

[0149] in, This indicates the number of time slots between two adjacent edge-side decision moments. Indicates the first The time slot offset index within each edge-side decision interval. This interval report is used to evaluate the cumulative impact of edge-side authorization and resource allocation actions on communication energy consumption, service reliability, and control performance within their effective time interval. Therefore, the... The edge-side semi-Markov transition samples generated at each edge-side decision time are represented as follows:

[0150] ;

[0151] in, Indicates the first The dynamic decision-making state of the edge side at each edge-side decision-making moment. This indicates the dynamic decision-making state of the edge side at the next edge-side decision-making moment; the dynamic decision-making state of the edge side consists of the state of each radio control loop in the current resource coordination set and the state of the currently available radio resources.

[0152] S23: Establish an edge decision-making coupling mechanism

[0153] Based on the end-side semi-Markov decision model established in step S21 and the edge-side semi-Markov decision model established in step S22, an interactive coupling mechanism is established between the two to form a complete hierarchical semi-Markov decision model.

[0154] When a wireless control loop arrives at the edge-side decision time, it determines the edge-side decision action based on the current edge-side local state. When the edge-side decision action indicates initiating an edge-assisted control request, the corresponding wireless control loop is added to the new arrival request set. When the edge server receives a new edge-assisted control request, it constructs the edge-side dynamic decision state based on the new arrival request, the in-service control loop, and available wireless resources, and performs new access authorization and wireless resource allocation. The access authorization result and resource allocation result determine whether the state upload can be completed within the specified time delay, and further determine whether effective edge control is formed. The effective edge control result changes the control input adopted by the controlled object, causing changes in the state of the controlled object and the control error; after the sensor re-acquires the state, it updates the edge-side local state and the edge-side dynamic decision state.

[0155] In the aforementioned interaction process, the endpoint decision-making actions influence edge-side access authorization selection and radio resource allocation by altering the edge-assisted control request set. Edge-side access authorization selection and radio resource allocation further influence subsequent endpoint local states and endpoint decisions by changing state upload results, effective edge control results, and the state of the controlled object. Therefore, the endpoint-side semi-Markov decision model and the edge-side semi-Markov decision model are interactively coupled through edge-assisted control requests, access authorization results, transmission success status, effective edge control indicator variables, and controlled object state transitions, jointly constituting a hierarchical semi-Markov decision model.

[0156] S3: Design a shared decision network on the edge side and a tokenized decision network on the edge side, and train them for edge-edge collaborative reinforcement learning based on a hierarchical semi-Markov decision model.

[0157] like Figure 3 As shown, this embodiment constructs an edge-side shared decision network and an edge-side tokenized decision network. The edge-side shared decision network is deployed in each wireless control loop. Each wireless control loop uses the same network structure and shared parameters, and independently generates edge-side decision actions based on its own local edge state. The edge-side tokenized decision network is deployed on the edge server to handle dynamically changing resource coordination sets and output access authorization and wireless resource allocation results. The two types of networks do not directly share hidden features or online decision parameters, but interact through edge-assisted control requests, access authorization results, state upload results, and effective edge control indicator variables.

[0158] S31: Construct an edge-side shared decision-making network;

[0159] Since each wireless control loop has the same end-side task structure, this embodiment uses a parameter sharing mechanism to construct an end-side shared decision network. Multiple wireless control loops are configured with parameter copies of the end-side shared decision network, each parameter copy having the same network structure and network parameters, and generating end-side decision actions based on the end-side local state of its corresponding wireless control loop.

[0160] The endpoint shared decision network uses the endpoint local state The input is the candidate action set on the endpoint, and the output is the indicator variable. , It contains 2 Each candidate edge decision action is assigned a value, and the edge decision network outputs the action value corresponding to each candidate edge decision action, and selects the candidate action with the highest action value.

[0161] In one implementation, the edge-side shared decision network employs a dual deep Q-network based on a competitive structure, comprising an evaluation Q-network and a target Q-network. The evaluation Q-network outputs the current action value of each candidate edge-side decision action based on the current edge-side local state, while the target Q-network calculates the target action value during the training phase. The competitive structure decomposes the action value function into state values. and action advantages The action value of each candidate action is expressed as:

[0162] ;

[0163] in, This represents the set of candidate joint decision-making actions on the endpoint. This indicates the number of actions contained in the set. This indicates the action index used when traversing the action set. This represents the trainable parameters for evaluating the Q-network. This represents the evaluation of the Q-network's action value estimation for state-action pairs. The state value branch does not directly output a manually set "urgency level," but rather obtains the overall state value of the current edge-side local state through training, without distinguishing specific actions. This state value reflects the overall edge-assisted control demand formed by the current control error, safety margin, and communication status. The action advantage branch characterizes the superiority of different edge-assisted control request-duration slot joint actions relative to the average action under the same edge-side local state.

[0164] During the training phase, dual Q-learning is used to calculate the value of the target action. First, the evaluation Q-network selects the optimal candidate action corresponding to the next end-side local state:

[0165] ;

[0166] The semi-Markov objective value is then calculated using the target Q-network:

[0167] ;

[0168] in, The parameters of the target Q-network are defined here. Action selection and target value calculation are performed by different networks, thereby reducing overestimation caused by directly taking the maximum target Q-value. During the training phase, the evaluation Q-network is updated based on the target action value, and the target Q-network is synchronized with the parameters of the evaluation Q-network.

[0169] In one implementation, at least one fully connected layer of the end-side shared decision network is a noisy linear layer. The weights and biases of the noisy linear layer are expressed as follows:

[0170] ;

[0171] ;

[0172] in, , , and All are learnable parameters. and The noise is random. During training, the random noise is resampled at a preset frequency, causing the network to generate different action value perturbations in different local states at different endpoints, thereby realizing state-related action exploration; in the deterministic execution phase, the learnable mean parameter is used to output the action value.

[0173] S32: Construct an edge-side tokenized decision-making network;

[0174] The edge server is used for centralized coordination of edge-assisted control requests from multiple wireless control loops. Since the number of edge-assisted control requests arriving at the edge server varies at different decision times, and some wireless control loops may still be in the edge service process, using the state of all control loops with a fixed dimension as input would cause the state dimension to increase with the number of control loops, introducing a large amount of information unrelated to current resource contention. Therefore, this embodiment constructs a dynamic token set based on the wireless control loops currently participating in resource coordination.

[0175] In the Each edge decision moment Based on the current resource coordination set Construct a dynamic token set. For any... Construction and the first The edge-side token corresponding to each wireless control loop:

[0176] ;

[0177] in, Indicates the first A wireless control loop at the edge-side decision-making time The corresponding end-side local state, Indicates the first An edge control request indicator variable for a wireless control loop Indicates the first Is each wireless control loop in edge service mode? Indicates the first The number of remaining service slots for each wireless control loop. This indicates the decision-making time of the edge server at the edge. The number of remaining available subcarriers.

[0178] The edge-side dynamic token set is represented as:

[0179] ;

[0180] The edge-side dynamic token set is used as input to the edge-side tokenized decision network.

[0181] The edge-side tokenization decision network adopts an actor-critic structure, including an actor network and a critic network. The actor network includes an embedding module, an encoding module, an authorization selection module, and a resource allocation module, used to generate access authorization results and radio resource allocation results. The critic network is used to estimate the state value of the edge-side dynamic decision state based on the edge-side dynamic token set during the training phase. The embedding module first normalizes the continuous features in the edge-side tokens, and then maps the edge-side tokens to initial hidden features of a unified dimension through at least one fully connected layer. The encoding module includes at least one Transformer encoding block, each Transformer encoding block including a multi-head attention sublayer and a feedforward neural network sublayer, with each sublayer employing layer normalization and residual connections. Through multi-head self-attention computation, the hidden features of each radio control loop can adaptively fuse the request state, control state, and resource occupancy state of other radio control loops in the current resource coordination set, thereby obtaining relationship enhancement features that can characterize the resource competition relationship between loops. The set of relationship enhancement features after Transformer encoding is represented as follows:

[0182] ;

[0183] in, This represents the encoding function jointly constituted by the embedding module and the encoding module. Indicates the first Enhanced relationship features of a wireless control loop after incorporating information from other participating resource coordination loops.

[0184] The authorization selection module only applies to newly arriving request sets. The wireless control loop in the process performs a new authorization determination and outputs an authorization priority score based on the relationship enhancement feature:

[0185] ;

[0186] The number of new grants is determined based on the number of new arriving requests and the number of remaining available subcarriers:

[0187] ;

[0188] Based on the authorization priority score and the number of new access authorizations, the new authorization set is obtained:

[0189] ;

[0190] in, This represents the mapping function by which the authorization selection module outputs authorization priority scores based on relationship enhancement features; This indicates that the request set arrives again in descending order of authorization priority score. Before the election Operation of a wireless control loop.

[0191] After obtaining the newly added access authorization set, merge the newly added authorization set with the set of radio control loops currently receiving edge services to obtain the current set of radio control loops participating in radio resource allocation:

[0192] ;

[0193] The resource allocation module determines the set of wireless control loops currently participating in the wireless resource allocation. The relationship between each wireless control loop is enhanced, and the allocation parameters corresponding to the output bandwidth allocation random variable and transmit power allocation random variable are obtained. After normalization and resource constraint mapping, the actual bandwidth and transmit power allocation results are obtained.

[0194] S33: Training an edge-cooperative decision-making network based on semi-Markov transition samples;

[0195] During the training phase, the centralized training module maintains the shared decision network parameters on the endpoint side, the tokenized decision network parameters on the edge side, the endpoint experience cache, and the edge trajectory cache. During training interaction, each wireless control loop generates endpoint decision actions based on its own local endpoint state, and the edge server generates access authorization results, bandwidth allocation results, and transmit power allocation results based on the edge-side dynamic token set. The system advances the controlled object state and wireless communication state based on the endpoint and edge decision actions, and calculates endpoint and edge interval rewards respectively. When an endpoint decision action finishes execution, a corresponding endpoint semi-Markov transition sample is generated and stored in the endpoint experience cache. When the next edge-side decision time is reached, a corresponding edge semi-Markov transition sample is generated and stored in the edge trajectory cache.

[0196] For the edge-side shared decision network, edge-side semi-Markov transition samples are sampled from the edge-side experience cache, and the evaluation Q network is updated according to the target action value in step 31. The target Q network is synchronized by the evaluation Q network parameters.

[0197] For the edge-side tokenized decision network, the edge-side temporal difference residual is calculated based on the edge-side semi-Markov transition samples:

[0198] ;

[0199] in, This represents the state value of the commentator's network output. This represents the parameters of the critic network. The edge-side dominance estimate is calculated based on the edge-side temporal difference residuals, and the edge-side tokenized decision network is updated based on the edge-side dominance estimate. Wherein, the discount term... Used to reflect the variable time slot interval between two adjacent edge side decision moments.

[0200] The edge-side shared decision network and the edge-side tokenized decision network update their parameters based on their respective semi-Markov transition samples, but they do not directly share network parameters or training gradients. Edge-side decision actions affect the edge-side decision state by changing the edge-assisted control request set, and the edge-side access authorization and wireless resource allocation results affect subsequent edge-side decisions by changing the communication results and the state of the controlled object, thereby achieving edge-side collaborative training in a shared system interaction process.

[0201] S34: Output the edge collaborative decision-making strategy after training is complete.

[0202] Repeat the edge-to-edge interactive sampling and network update process in step S33 until the preset maximum number of training rounds is reached. After training, copies of the evaluation Q network parameters from the end-side shared decision network are deployed to each radio control loop, and the edge-side actor network parameters are deployed to the edge server. The trained end-side shared decision network determines whether to initiate an edge-assisted control request and the corresponding action duration slots based on the end-side local state of the radio control loop. The trained edge-side tokenized decision network executes access authorization based on the dynamic request set and generates bandwidth and transmit power allocation results that satisfy radio resource constraints.

[0203] During system operation, each wireless control loop selects an edge-assisted control request and action duration based on its own local state, and then performs a joint action. The edge server constructs a dynamic token set based on the set of newly arriving requests and the set of serving control loops, performs deterministic Top-K access authorization, and generates resource allocation results that satisfy system bandwidth and maximum transmit power constraints. Therefore, the wireless control loops do not need to repeatedly execute request decisions in each time slot, and the edge server does not need to construct fixed-dimensional states for all wireless control loops and perform global combinatorial optimization.

[0204] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for end-edge cooperative access control and resource allocation in a multi-loop wireless network control system, characterized in that, The steps are as follows: S1: Construct an end-edge collaborative control-communication architecture for a multi-loop wireless networked control system; S2: Model the end-side collaborative access control and resource allocation problem in the end-side collaborative control-communication architecture as a hierarchical semi-Markov decision model; S3: Design a shared decision network on the edge side and a tokenized decision network on the edge side, and conduct edge-edge collaborative reinforcement learning training based on a hierarchical semi-Markov decision model; The trained end-side shared decision network determines whether to initiate an edge-assisted control request and the corresponding duration of time slots based on the end-side local state of the wireless control loop. The trained edge-side tokenized decision network executes access authorization based on the dynamic request set and generates bandwidth and transmit power allocation results that satisfy wireless resource constraints.

2. The end-edge cooperative access control and resource allocation method for a multi-loop wireless network control system according to claim 1, characterized in that, The edge-coordinated control-communication architecture includes There are one wireless control loop and one edge server. The set of wireless control loops is denoted as . Divide the runtime into A time slot of equal length, time slot index ; in each time slot Within the system, each wireless control loop determines whether to request edge-assisted control based on its own local control state and wireless communication state, thereby determining whether to generate an edge-assisted control request. The edge server performs access authorization selection, bandwidth allocation, and transmit power allocation based on the received edge-assisted control request, the current wireless resource status, and the status of the control loop being served.

3. The end-edge cooperative access control and resource allocation method for a multi-loop wireless network control system according to claim 1 or 2, characterized in that, The hierarchical semi-Markov decision model includes an end-side semi-Markov decision model and an edge-side semi-Markov decision model. The end-side semi-Markov decision model is used to describe how each wireless control loop determines whether to request edge-assisted control based on the end-side local state, and the number of time slots that need to be maintained for the edge-assisted control request decision. The edge-side semi-Markov decision model is used to describe how the edge server performs centralized resource coordination based on dynamically arriving edge-assisted control requests and the current wireless resource status. The edge-side semi-Markov decision model selects licenses and allocates resources based on the set of wireless control loops currently competing for resources. The interaction coupling mechanism between the end-side semi-Markov decision model and the edge-side semi-Markov decision model: When the wireless control loop arrives at the edge-side decision time, it determines the edge-side decision action based on the current edge-side local state. When the edge-side decision action indicates initiating an edge-assisted control request, the corresponding wireless control loop is added to the new arrival request set. When the edge server receives a new edge-assisted control request, it constructs the edge-side dynamic decision state based on the new arrival request, the in-service control loop, and available wireless resources, and performs new access authorization and wireless resource allocation. The results of the new access authorization and wireless resource allocation determine whether the state upload can be completed within the specified time delay, and further determine whether effective edge control is formed. The effective edge control result changes the control input adopted by the controlled object, causing changes in the state and control error of the controlled object. After the sensor re-acquires the state, it updates the edge-side local state and the edge-side dynamic decision state.

4. The end-edge cooperative access control and resource allocation method for a multi-loop wireless network control system according to claim 3, characterized in that, The edge-side decision-making actions affect the selection of new access authorizations and the allocation of radio resources on the edge side by changing the set of edge-assisted control requests; the selection of new access authorizations and the allocation of radio resources on the edge side further affect the subsequent local state and edge-side decision-making actions by changing the state upload results, the effective edge control results, and the state of the controlled object. No. A wireless control loop at the end-side decision time The resulting end-side semi-Markov transfer samples are: ； in, Indicates the first The wireless control loop in the first Local state of the endpoint at each endpoint decision moment Indicates the next decision-making moment. The local state of the end side, For edge-side decision-making actions, This indicates the number of time slots during which the decision-making action is performed on the endpoint. For the first The first wireless control loop Individual decision-making moment Range of returns; No. The edge-side semi-Markov transition samples generated at each edge-side decision time are: ； in, Indicates the first The dynamic decision-making state of the edge side at each edge-side decision-making moment. Indicates the first The dynamic decision-making state of the edge side at each edge-side decision-making moment; For the first Actions executed at the edge of the decision-making process; For the first Edge-side interval return at each edge-side decision moment; This represents the number of time slots between two adjacent edge-side decision moments.

5. The end-edge cooperative access control and resource allocation method for a multi-loop wireless network control system according to claim 4, characterized in that, The end-side local state of the end-side semi-Markov decision model includes local channel state information, waiting time, safety margin, and safety margin change trend; The local state at the end of each wireless control loop is represented as follows: ； in, Indicates the first A wireless control loop in the time slot The obtained local channel state information, For the first A wireless control loop in the time slot Waiting time, For the first A wireless control loop in the time slot Safety margin, For safety margin Trends; No. Each wireless control loop is based on the local state of the end side. The output edge-side decision action is a joint action of request and duration slot number: ;in, Indicates the first The wireless control loop in the first Individual decision-making moment An indicator variable for whether to initiate an edge-assisted control request. Representation and indicator variables The corresponding number of time slots for edge-side decision-making actions. Maximum duration; No. The first wireless control loop Individual decision-making moment The range of returns is: ； in, This represents the decision cost incurred by the endpoint in executing a decision once at a single decision moment. The discount factor; the first A wireless control loop in the time slot End-side instant feedback ； in, Indicates the first A wireless control loop in the time slot Tracking error Relative to threshold Safety margin, The energy consumed to initiate an edge-assisted control request. For the first A wireless control loop in the time slot Edge-assisted control request variables; Indicates the first A wireless control loop in the time slot Whether edge-assisted control is effectively implemented. For the first A wireless control loop in the time slot Effective edge control indicator variables, Indicates tracking error Exceeding the threshold Indicator functions, , , , All are positive weighting coefficients; The edge-side dynamic decision state is determined by the first... Each edge decision moment The current set of wireless control loops involved in authorization determination and resource preservation The status of each wireless control loop and the current status of available wireless resources constitute the basis for... ;in, Indicates the first The status of each wireless control loop For the first A wireless control loop at the edge-side decision-making time Are there active edge-assisted control request variables? Indicates the first The current service status of each wireless control loop. Indicates the first A wireless control loop at the edge-side decision-making time The number of remaining service slots; Indicates the decision-making moment at the edge. The number of remaining subcarriers that can be used to add new access licenses; No. Actions executed at the edge of the decision-making process. ;in, and They represent the first time. The edge decision time is assigned to the first Bandwidth and transmit power of each wireless control loop; For edge servers in the first Each edge decision moment The newly arrived edge-assisted control requests form a new arrival request set. Select the new access authorization set; No. Each edge decision moment The edge-side interval return is: ;in, 1 indicates the first Slot offset index within each edge-side decision moment; edge-side instant report ;in, Indicates the first A wireless control loop in the time slot The transmission power, Indicates the maximum permissible transmission power; for Uplink latency of each wireless control loop Indicates the first A wireless control loop in the time slot The scheduling variable; This indicates a penalty for edge service failure. , , All are positive weighting coefficients.

6. The end-edge cooperative access control and resource allocation method for a multi-loop wireless network control system according to claim 4 or 5, characterized in that, The end-side shared decision network is deployed in each wireless control loop. The end-side shared decision network of each wireless control loop uses the same network structure and network parameters, and independently generates end-side decision actions based on their respective end-side local states. The edge-side tokenized decision network is deployed on the edge server to process resource coordination sets with dynamically changing scales and output access authorization and wireless resource allocation results. The end-side shared decision network and the edge-side tokenized decision network interact through edge-assisted control requests, access authorization results, status upload results, and valid edge control indicator variables. The edge-side tokenization decision network constructs a dynamic token set for the edge side based on the wireless control loops currently participating in resource coordination; Training end-side shared decision networks and edge-side tokenized decision networks using semi-Markov transfer samples from the end and edge sides.

7. The end-edge cooperative access control and resource allocation method for a multi-loop wireless network control system according to claim 6, characterized in that, The endpoint shared decision network uses the endpoint local state As input, the candidate actions on the endpoint are determined by the indicator variable. and Combining and consisting of 2 Each candidate edge-side decision action is assigned a value corresponding to a specific action in the shared decision network output. The candidate edge-side decision action with the highest value is then selected. In the Each edge decision moment According to the wireless control loop that participates in authorization judgment and resource maintenance Construct a dynamic token set for the edge side: for any Construction and the first The edge-side token corresponding to each wireless control loop: ； in, Indicates the first A wireless control loop at the edge-side decision-making time The corresponding end-side local state, Indicates the first An edge control request indicator variable for a wireless control loop Indicates the first Is each wireless control loop in edge service mode? Indicates the first The number of remaining service slots for each wireless control loop. This indicates the decision-making time of the edge server at the edge. The number of remaining available subcarriers; The edge-side dynamic token set is represented as: ; The end-side shared decision network and the edge-side tokenized decision network update parameters based on the end-side semi-Markov transition samples and the edge-side semi-Markov transition samples, respectively. End-side decision actions affect the edge-side decision state by changing the edge-assisted control request set. Edge-side access authorization and wireless resource allocation results affect subsequent end-side decisions by changing communication results and the state of the controlled object. End-side collaborative training is achieved in the common interaction process. During the training phase, the parameters of the end-side shared decision network, the parameters of the edge-side tokenized decision network, the end-side experience cache, and the edge-side trajectory cache are maintained. During the training interaction, each wireless control loop generates end-side decision actions based on its own end-side local state, and the edge server generates access authorization results, bandwidth allocation results, and transmit power allocation results based on the edge-side dynamic token set. The state of the controlled object and the wireless communication state are advanced according to the end-side decision actions and the edge-side decision actions, and the end-side interval reward and the edge-side interval reward are calculated respectively. When the end-side decision action is completed, the corresponding end-side semi-Markov transition sample is generated and stored in the end-side experience cache. When the next edge-side decision time is reached, the corresponding edge-side semi-Markov transition sample is generated and stored in the edge-side trajectory cache.

8. The end-edge cooperative access control and resource allocation method for a multi-loop wireless network control system according to claim 7, characterized in that, The edge-side shared decision network employs a dual-depth Q-network based on a competitive structure, comprising an evaluation Q-network and a target Q-network. The evaluation Q-network outputs the current action value of each candidate edge-side decision action based on the current local state of the edge, while the target Q-network calculates the target action value during the training phase. The competitive structure decomposes the action value function into state values. and action advantages The current action value of each candidate endpoint decision action is: ； in, Represents the joint set of candidate decision actions on the endpoint. Number of actions included Represents the set of action unions The action index used during traversal; This represents the trainable parameters for evaluating the Q-network; This indicates the evaluation of the Q-network's estimation of the action value of state-action pairs; The edge-side tokenization decision network adopts an actor-commentator structure, including an actor network and a commentator network. The actor network includes an embedding module, an encoding module, an authorization selection module, and a resource allocation module, used to generate access authorization results and radio resource allocation results. The commentator network is used to estimate the state value of the edge-side dynamic decision state based on the edge-side dynamic token set during the training phase. The embedding module of the actor network normalizes the continuous features in the edge-side tokens, and then maps the edge-side tokens to initial hidden features of a unified dimension through at least one fully connected layer. The encoding module includes at least one Transformer encoding block, each Transformer encoding block including a multi-head attention sublayer and a feedforward neural network sublayer. Both the multi-head attention sublayer and the feedforward neural network sublayer employ layer normalization and residual connections. Through multi-head self-attention computation of the multi-head attention sublayer, the hidden features of each radio control loop adaptively fuse with the current resource-coordinated radio control loop set. The request status, control status, and resource occupancy status of other wireless control loops are analyzed to obtain enhanced relationship features characterizing the resource contention relationship between loops; the authorization selection module only considers newly arriving request sets. The radio control loop in the system performs a new license determination and outputs a license priority score based on the relationship enhancement features. It determines the number of new licenses based on the number of new arrival requests and the number of remaining available subcarriers. Finally, based on the license priority score and the number of new access licenses, it obtains a set of new access licenses. ; Add new access authorization sets With the set of wireless control loops currently in service By merging, we obtain the set of wireless control loops currently participating in wireless resource allocation. The resource allocation module determines the set of wireless control loops currently participating in wireless resource allocation. The relationship between each wireless control loop is enhanced, and the allocation parameters corresponding to the output bandwidth allocation random variable and transmit power allocation random variable are obtained. After normalization and resource constraint mapping, the actual bandwidth and transmit power allocation results are obtained.

9. The end-edge cooperative access control and resource allocation method for a multi-loop wireless network control system according to claim 8, characterized in that, During the training phase, dual Q-learning is used to calculate the target action value. End-side semi-Markov transition samples are sampled from the end-side experience cache. The evaluation Q-network is updated based on the target action value, and the target Q-network is synchronized with the parameters of the evaluation Q-network. The evaluation Q-network selects the optimal candidate end-side decision action corresponding to the next end-side local state. ; Calculate the value of a semi-Markov objective action using a target Q-network: ;in, The network parameters of the target Q-network; The set of relationship-enhanced features encoded by the Transformer coding block is as follows: ； in, This represents the encoding function jointly constituted by the embedding module and the encoding module. Indicates the first The set of dynamic tokens on the edge side at each edge-side decision moment. Indicates the first Enhanced relationship features of a wireless control loop after fusing information from other participating resource coordination loops; Authorization priority score: ; The number of new authorizations is: ; The newly added access authorization set is: ; in, This represents a mapping function that outputs an authorization priority score based on relation enhancement features; This indicates that the request set arrives again in descending order of authorization priority score. Before the election Operation of a wireless control loop; For the edge-side tokenized decision network, the edge-side temporal difference residual is calculated based on the edge-side semi-Markov transition samples: ； in, This represents the state value of the commentator's network output. This represents the parameters of the critic network. This is a discount term; the edge-side dominance estimate is calculated based on the edge-side temporal difference residual, and the edge-side tokenized decision network is updated based on the edge-side dominance estimate.

10. The end-edge cooperative access control and resource allocation method for a multi-loop wireless network control system according to claim 8 or 9, characterized in that, The edge server based on the newly arrived request set and the set of wireless control loops currently in service Construct the set of wireless control loops currently involved in authorization determination and resource preservation: ; Edge servers in the Each edge decision moment The set of wireless control loops corresponding to the newly arrived edge-assisted control requests. Select Add Access Authorization Set It also allocates bandwidth and transmit power to newly granted access and wireless control loops that are receiving edge-assisted control. The newly added access authorization set satisfies: ;in, Indicates the newly added access authorization set The number of wireless control loops in the middle; At least one fully connected layer of the end-side shared decision network is a noisy linear layer, and the weights and biases of the noisy linear layer are expressed as follows: and ;in, , , and All are learnable parameters. and The noise is random; during training, random noise is resampled at a preset frequency so that the edge-side shared decision network generates different action value perturbations in different edge-side local states, thereby realizing state-related action exploration; during the deterministic execution phase, learnable parameters are used to output action values. Repeat the edge-to-edge interactive sampling and network update process until the preset maximum number of training rounds is reached. After training is completed, the evaluation Q network parameter copies in the edge-side shared decision network are deployed to each wireless control loop, and the actor network parameters of the edge-side tokenized decision network are deployed to the edge server. During operation, each wireless control loop determines whether to initiate an edge-assisted control request and the corresponding duration of action slots based on its own local end-side state. The edge server constructs an edge-side dynamic token set based on the set of newly arrived requests and the set of wireless control loops currently being served, performs deterministic Top-K access authorization, and generates resource allocation results that satisfy system bandwidth and maximum transmit power constraints.

Images