Industrial wireless network transmission and computing resource allocation method under imperfect channel
By designing an end-edge collaborative computing architecture based on D2D assisted learning and a multi-agent deep reinforcement learning framework in industrial wireless networks, combining Transformer and ACK feedback recovery mechanisms, the real-time and stability of heterogeneous tasks under imperfect channels is solved. The problem has achieved the improvement of stability and adaptability of task processing.
Patent Information
- Application Number
- CN202411063971.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-05
AI Technical Summary
In industrial wireless networks, the real-time requirements of heterogeneous tasks are difficult to meet, especially in the case of time-varying equipment number and imperfect channels, the existing multi-agent reinforcement learning algorithms are insufficient in generalization and difficult to adapt to noise interference. , affects the stability and real-time nature of task processing.
A multi-agent deep reinforcement learning framework based on D2D assisted learning was designed, and a multi-agent deep reinforcement learning framework combining Transformer and ACK feedback recovery mechanism. Through the distributed part, Markov decision-making process and feedback recovery mechanism are observed, the Markov decision-making process and feedback recovery mechanism are realized. Joint task offloading and multi-channel access algorithms under imperfect channels improve the stability and adaptability of the system.
This method improves the total delay minimization capability of task processing in industrial wireless networks, enhances the security and stability of the system, adapts to time-varying topological scenarios and harsh communication environments, and improves the generalization and robustness of the algorithm.
Smart Images

Figure CN119071926A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of transmission and computing resource allocation in industrial wireless networks, and in particular to an industrial wireless network transmission and computing resource allocation method under an imperfect channel. Background Art
[0002] With the development of intelligent manufacturing, industrial equipment such as automated guided vehicles (AGVs), robotic arms, and mobile robots are widely used in smart factories. These devices are interconnected through industrial wireless networks and generate massive heterogeneous tasks with different deadlines and computing requirements. How to meet the real-time requirements of heterogeneous tasks under the limited communication and computing resources of industrial wireless networks has become a challenge.
[0003] Through multi-access edge computing technology, industrial equipment can offload heterogeneous tasks to edge servers for computing to reduce task processing latency. Previous work mainly reduced the total latency of heterogeneous task processing by optimizing task offloading strategies. However, in industrial wireless networks, massive heterogeneous tasks need to compete for limited channel resources when performing high-concurrency access, which will lead to access conflicts and affect the real-time processing of tasks. Therefore, in order to meet the real-time requirements of heterogeneous tasks in smart factories, it is necessary to jointly optimize task offloading and multi-channel access strategies. However, the heterogeneity of the tasks generated by the devices makes the offloading requirements of the devices time-varying and difficult to accurately model. In addition, since distributed deployed devices cannot obtain global information, this will aggravate the conflicts of multi-channel access of devices. Multi-agent reinforcement learning algorithms can model devices as agents, alleviate the non-stationary scenarios caused by limited local observation of devices, and approximate the optimal strategy through the interaction between devices and the environment without the need for accurate modeling. Therefore, multi-agent reinforcement learning algorithms are gradually being used to solve the problems of joint task offloading and multi-channel access.
[0004] However, the current algorithm model still has the following two problems that hinder its application in actual industrial scenarios. First, in actual industrial scenarios, the number of devices is time-varying. The current algorithm is not general enough. Since the input dimension of the neural network is bound to the number of devices, once the number of devices changes, the algorithm model needs to change the input dimension and start training from scratch. Second, the training of multi-agent reinforcement learning algorithms requires the interaction between devices and the environment to generate feedback signals. Current algorithms often assume that devices can always receive correct feedback from interactions with the environment. However, the existence of imperfect channels with strong noise interference in industrial wireless networks causes feedback loss, which in turn affects the normal training of devices.
[0005] In order to make up for the shortcomings of the above joint task offloading and multi-channel access algorithm and make the algorithm more suitable for actual industrial scenarios, it is necessary to design a joint task offloading and multi-channel access method with better generalization and coping with imperfect channels to allocate the transmission and computing resources of industrial wireless networks. Summary of the invention
[0006] In view of the technical problem of poor stability of transmission and computing resource allocation in industrial wireless networks, the present invention proposes an industrial wireless network transmission and computing resource allocation method under imperfect channels, which has high system security and stability.
[0007] In order to achieve the above object, the technical solution of the present invention is implemented as follows: an industrial wireless network transmission and computing resource allocation method under an imperfect channel, characterized in that the steps include:
[0008] S1: Establishing an edge-to-edge collaborative computing architecture for D2D-assisted learning in industrial wireless networks with imperfect channels;
[0009] S2: Considering the different real-time and computing resource requirements of heterogeneous tasks, we combine multi-channel access and task offloading to obtain a distributed partially observable Markov decision process with a graph structure.
[0010] S3: Based on a multi-agent deep reinforcement learning framework that integrates Transformer and ACK feedback recovery mechanism, a migratable joint task offloading and multi-channel access algorithm is designed to cope with imperfect channels based on the edge-end collaborative computing architecture, and a computing resource allocation method is obtained. Industrial equipment makes task offloading and multi-channel access decisions based on its own observations to minimize the total delay of task processing.
[0011] The edge-end collaborative computing architecture described in step S1 includes an industrial base station, an edge server, and M industrial devices. The edge server corresponds to the industrial base station. The industrial devices are connected to the edge server through a wireless network. The industrial devices communicate with each other through D2D and generate heterogeneous tasks. The heterogeneous tasks generated by industrial device m at time slot t are d m,n , c m,n , They represent the task size, the required computing delay and the maximum tolerable delay respectively. Represents task n m (t) is a collection of
[0012] The method for obtaining the distributed partially observable Markov decision process of the graph structure in step S2 is:
[0013] The observable Markov decision process includes graph structure observation, action and reward function. Each industrial device is set as an entity with z features. The observation of industrial device m includes its own features and m-1 features of other entities. The feature information of other entities is obtained through D2D communication.
[0014] At time slot t, the entity i, i∈M observed by industrial equipment m has its characteristics defined as follows:
[0015] ent m,i (t) = [ack i (t),d i,n (t),c i,n (t),h i,n (t)];
[0016] Among them, ack i (t) indicates the base station confirmation feedback received by entity i, ack i (t)={null,-1,0,1},ack i (t) = null means ack feedback is lost, ack i (t) = -1 indicates that the task transmission fails, ack i (t) = 0 means the task is offloaded locally, ack i (t) = 1 means the task is successfully completed, d i,n (t), c i,n (t), h i,n (t) represents the task n generated by entity i m (t) is the task size, required computing resources and remaining lifetime, n represents task n m (t) index;
[0017] At time slot t, the graph observation space o of industrial equipment m m (t) is an m*z matrix, expressed as:
[0018] o m (t) = [ent m,1 (t),ent m,2 (t),...,ent m,i (t)];
[0019] The overall observation space of the system can be expressed as:
[0020] o(t)={o1(t),o2(t),...,o M (t)};
[0021] Among them, ent m,i (t) is the physical feature observed by the industrial equipment, o M(t) is the graph structure observation, and the channel selection action a of industrial equipment m at time slot t m (t) is defined as follows:
[0022] a m (t) = {0, 1, ..., K};
[0023] when a m When (t) = 0, it means that industrial equipment m chooses to offload tasks locally. m When (t) = k, it means that industrial equipment m selects channel k for task offloading; k∈K;
[0024] The reward function is designed by comprehensively considering the task processing delay optimization and task overdue penalty.
[0025] The method for designing a reward function by comprehensively considering task processing delay optimization and task overdue penalty is:
[0026] In order to minimize the processing delay of tasks, industrial equipment needs to offload tasks to edge servers for calculation. Therefore, the delay reward d for industrial equipment m at time slot t is m (t) is expressed as:
[0027]
[0028] in, For task n m (t) local processing delay at industrial device m, Represents task n m (t) the processing delay offloaded by industrial device m to the edge server;
[0029] In order to reduce the overdue rate of tasks, the overdue penalty p of industrial equipment m at time slot t is m (t) is expressed as
[0030]
[0031] in, For task n m (t) queuing delay in the buffer of industrial equipment m;
[0032] In summary, at time slot t, the compound reward function r of industrial equipment m is m (t) can be expressed as:
[0033] r m (t) = d m (t)-α·p m (t);
[0034] Among them, α is the weight parameter;
[0035] When the feedback of industrial equipment m is lost, the compound reward function r m (t) is set to null.
[0036] It is characterized in that the method of designing a joint task offloading and multi-channel access algorithm that can be migrated in the case of imperfect channels in step S3 is:
[0037] S31: Build an Agent network deployed on industrial equipment;
[0038] S32: Build a Mixing network deployed on edge servers;
[0039] S33: Design a feedback recovery mechanism to cope with ACK feedback loss caused by imperfect channels;
[0040] S34: A joint task offloading and multi-channel access algorithm that can be transferred to cope with imperfect channels is obtained. During the training period, each industrial device observes the output action according to the graph structure and obtains rewards. The industrial wireless network determines the value of the output action based on the rewards and performs network updates.
[0041] The method of building an Agent network deployed on industrial equipment in step S31 is:
[0042] The core architecture of the Agent network consists of the Embedding layer, Transformer module, and output layer connected in sequence;
[0043] The Embedder layer first creates a matrix Emb with a shape of z*h m Observation of graph structure m (t) is encoded, h represents the embedding dimension of the Embedder layer, and z represents the matrix Embed m dimension, the final encoded matrix E(t) can be obtained by the formula E(t) = o m (t)Emb m Calculated; o m The same features in (t) are in the matrix Emb m share the same weights, and the matrix Emb m It has nothing to do with the number of industrial equipment;
[0044] The input of the Transformer module includes the matrix E(t) and h m (t-1), where h m (t-1) is the hidden state containing the action and observation history information of industrial equipment m. The Transformer module processes the matrix E(t) and h in parallel through the multi-head self-attention mechanism. m (t-1), learn the edge relationships between entities in the input matrix and obtain the improved graph representation matrix Em (t) and h m (t), where the matrix E m The vertex vector of (t) fuses the information of adjacent related vertices; the output layer passes through h m (t) and E m (t) Information generates Q value Q m (τ m ,a m (t)), used to evaluate the quality of the current actions taken by industrial equipment, τ m Action-observation history information for industrial equipment.
[0045] The method of step S32 for building a Mixing network deployed on the edge server is:
[0046] The core architecture of the Mixing network consists of the Embedder layer, Transformer module and output layer connected in sequence. The output layer consists of two deep neural networks, namely the parameter generation network and the inference network.
[0047] The input of the Embedder layer of the Mixing network is the observation s(t) of all industrial equipment. The encoded observation s(t) is sent to the Transformer module for feature learning, and finally combined with the parameter generation network to generate neuron weights w1 and w2; the inference network generates a global Q value through neuron weights w1 and w2 and the Q value of all industrial equipment. tot (τ, a(t)), the optimization goal of the Agent network is to maximize its own Q value, and the optimization goal of the Mixing network is to minimize Q tot The mean square error between (τ,a(t)) and the target Q value, the Mixing network is constrained Ensure the consistency of the two optimization goals and learn the optimal network resource allocation strategy.
[0048] Step S33: A feedback recovery mechanism is designed to cope with ACK feedback loss caused by an imperfect channel, including:
[0049] S331: Model the channel in the edge-to-edge collaborative computing architecture as an imperfect channel industrial device with data loss probability, collect information about the interaction between the industrial device and the industrial wireless network, including observations, industrial device actions and rewards, including feedback loss and reward loss information to build a training data set;
[0050] S332: Construct the interaction into a data frame in pandas format and use the deep learning-based DataWig model to predict missing data;
[0051] S333: The ack signal and industrial equipment action in the edge server are classified and encoded using one-hot vectors, and feature extraction is performed using the Embedder layer. The remaining data is vector feature concatenated through a normalized neural network;
[0052] S334: Use the classification cross entropy loss function and the regression mean square error loss function to jointly train the classification target and the continuous numerical target respectively;
[0053] S335: Use the trained neural network model output prediction value to replace the missing feedback in the original data to restore the complete empirical data to cope with the challenges of imperfect channels to model training.
[0054] The classification target in step S334 includes the ack signal and the industrial equipment action, and the continuous numerical target includes the remaining observation values and the reward value.
[0055] The steps of the joint task offloading and multi-channel access algorithm for coping with imperfect channels in step S34 are as follows:
[0056] S341: Initialize the parameters of the neural network model and Transformer module of the algorithm;
[0057] S342: constructing a graph structure observation according to the D2D communication range between the industrial devices, the features of other industrial devices within the observation range of the industrial device m are added to the observation of the industrial device m, and the features of the industrial devices outside the observation range are filled with 0;
[0058] S343: The industrial equipment interacts with the industrial wireless network environment according to the graph structure observation output action and obtains rewards. If the feedback signal of the environment interaction is lost due to the interference of the imperfect channel, it is recovered by using the feedback recovery mechanism;
[0059] S344: Collect the complete observations, states, actions, and rewards of industrial equipment as experience trajectories and update the algorithm’s neural network;
[0060] S345: Repeat iterating S341-S344 until the number of simulation rounds is reached to complete the training. After the training is completed, each industrial device can perform adaptive task offloading and channel access decisions in a distributed manner to achieve reasonable allocation of transmission and computing resources in the industrial network;
[0061] The steps of the feedback recovery mechanism in step S344 are:
[0062] S3441: Model the channel as an imperfect channel device with ε data loss probability, collect information about the interaction between industrial devices and industrial wireless networks, including observations, device actions, and rewards, including feedback loss and reward loss information to construct a training data set;
[0063] S3442: Construct the interactions into a pandas data frame and use the deep learning-based DataWig model to predict missing data;
[0064] S3443: The ack signal and device action in the edge server are classified and encoded using one-hot vectors, and feature extraction is performed using the Embedder layer. The remaining data is vector feature concatenated through a normalized neural network;
[0065] S3444: Use the classification cross entropy loss function and the regression mean square error loss function to jointly train the ack signal and the device action and other observation values and reward values respectively;
[0066] S3445: Use the trained neural network model output prediction value to replace the missing feedback in the original data to restore the complete empirical data to cope with the challenges of imperfect channels to model training.
[0067] The beneficial effects of the present invention are:
[0068] 1. The present invention comprehensively considers the different real-time requirements and computing resources of heterogeneous tasks, and designs a joint task offloading and multi-channel access algorithm based on a data-driven multi-agent reinforcement learning algorithm. Without the need for precise system modeling, industrial equipment can be adaptively accessed based on interaction with the environment, and has high system security and stability.
[0069] 2. The present invention has strong generalization and practicality. The algorithm model can adapt to the industrial wireless network environment under time-varying topology scenarios, which makes up for the shortcomings of previous algorithms that need to be retrained after slight changes in topology.
[0070] 3. The present invention has strong robustness. Through the designed feedback recovery mechanism, the algorithm can adapt to industrial scenarios with harsh communication environments, thereby improving the performance and reliability of the algorithm in actual industrial applications. BRIEF DESCRIPTION OF THE DRAWINGS
[0071] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0072] Figure 1 An end-edge collaborative computing architecture for D2D-assisted learning is provided in accordance with an embodiment of the present invention.
[0073] Figure 2A schematic diagram of a joint task offloading and multi-channel access algorithm model based on multi-agent deep reinforcement learning provided by an embodiment of the present invention.
[0074] Figure 3 A flow chart of an industrial wireless network transmission and computing resource allocation method under an imperfect channel provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0075] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0076] like Figure 3 As shown, the present invention proposes a distributed multi-channel access method for multi-source heterogeneous cooperative transmission based on deep reinforcement learning, comprising the following steps:
[0077] Step 1: Establish an edge-to-end collaborative computing architecture for D2D-assisted learning in industrial wireless networks with imperfect channels;
[0078] like Figure 1 As shown in the figure, in order to alleviate the environmental non-stationarity caused by the difficulty of distributed devices in obtaining global information, an edge-end collaborative computing architecture suitable for D2D-assisted industrial wireless networks is first established.
[0079] The edge server is connected to the industrial base station through a wired network to meet the computing needs of heterogeneous tasks. The device can offload the generated tasks to the edge server through the industrial base station for computing. Industrial equipment includes distributed intelligent devices such as automatic guided vehicles (AGVs) and robotic arms. They are connected to the industrial base station through a wireless network and generate heterogeneous tasks. The device m, m∈M, generates heterogeneous tasks at time slot t. d m,n , c m,n , They represent the task size, required computation, and maximum tolerable delay, respectively. Represents task n m (t) is a collection of
[0080] Industrial equipment needs to make offloading decisions based on task attributes. When offloading heterogeneous tasks to edge servers, they need to compete for channels for transmission, which leads to problems of task offloading and multi-channel access. Due to the harsh communication environment in factories, there is a certain amount of noise interference in the channels. In the uplink transmission phase, when multiple devices choose the same channel for transmission or are interfered by channel noise, the task transmission fails. The failed task needs to wait for retransmission in the next time slot in the device's buffer in a first-in-first-out mode. In the downlink transmission phase, the ACK feedback signal of the base station will be lost when it is interfered, which affects the training process of the algorithm.
[0081] Step 2: Considering the different real-time and computing resource requirements of heterogeneous tasks, the joint multi-channel access and task offloading problem is formulated as a distributed partially observable Markov decision process with a graph structure.
[0082] Since distributed devices are in a partially observable state, each device can only observe its own access information and cannot obtain global information. In order to solve this problem, we first model the task offloading and multi-channel access problems of distributed devices into a distributed partially observable Markov decision model according to the following steps:
[0083] First, each industrial equipment is set as an entity with z features. The observation of industrial equipment m includes its own features and m-1 features of other entities. The feature information of other entities is obtained through D2D communication. The state space of the graph structure is conducive to the designed algorithm to better extract the device feature information.
[0084] At time slot t, the entity i, i∈M observed by device m has the following features:
[0085] ent m,i (t) = [ack i (t),d i,n (t),c i,n (t),h i,n (t)],
[0086] Among them ack i (t) = {null, -1, 0, 1} indicates the base station confirmation feedback received by entity i, ack i (t) = null means ack feedback is lost, ack i (t) = -1 indicates that the task transmission fails, ack i (t) = 0 means the task is offloaded locally, ack i (t) = 1 means the task is successfully completed, d i,n (t), c i,n (t), h i,n (t) represents the task n generated by entity i m(t) task size, required computing resources and remaining lifetime, etc.
[0087] At time slot t, the graph observation space o of device m m (t) is a matrix of m*z, which can be expressed as o m (t) = [ent m,1 (t),ent m,2 (t),...,ent m,i (t)],ent m,i (t) is the physical feature observed by the industrial equipment. The overall observation space of the system can be expressed as o(t) = {o1(t), o2(t), ..., o M (t)}.
[0088] Industrial device m observes the channel selection action a at time slot t m (t) is defined as follows:
[0089] a m (t)={0,1,...,K}.
[0090] when a m When (t) = 0, it means that industrial equipment m chooses to offload tasks locally. m When (t) = k, it means that industrial equipment m selects channel k (k∈K) for task offloading;
[0091] The reward function is designed by comprehensively considering the optimization of task processing delay and the penalty of task overdue, including the following steps:
[0092] In order to minimize the processing delay of the task, the device needs to offload the task to the mobile edge server for calculation. Therefore, the delay reward d of device m at time slot t is m (t) is expressed as
[0093]
[0094] in For task n m (t) local processing delay at device m, Task m (t) the processing delay offloaded by device m to the mobile edge server;
[0095] In order to reduce the overdue rate of tasks, the overdue penalty p of device m at time slot t is m (t) is expressed as:
[0096]
[0097] in For task n m (t) queuing delay in the buffer of device m;
[0098] In summary, at time slot t, the compound reward function r of device m is m (t) can be expressed as
[0099] r m (t) = d m (t)-α·p m (t), α is the weight parameter;
[0100] When the feedback from device m is lost, r m (t) is set to null.
[0101] Step 3: Based on the multi-agent deep reinforcement learning framework that integrates Transformer and ACK feedback recovery mechanism, a joint task offloading and multi-channel access algorithm that can be transferred to deal with imperfect channels is designed. Industrial equipment makes task offloading and multi-channel access decisions based on its own observations to minimize the total delay of task processing. The overall network architecture of the algorithm is as follows: Figure 2 shown.
[0102] First, build an Agent network deployed on the device. The core architecture of the Agent network consists of an Embedding layer, a Transformer module, and an output layer.
[0103] The Embedder layer first creates a matrix Emb with a shape of z*h m Observation of graph structure m (t) is encoded, h represents the embedding dimension of the Embedder layer, and the final encoded matrix E(t) can be obtained by the formula E(t) = o m (t)Emb m Calculated. m Similar features in (t) (such as task size of different entities, etc.) are m share the same weight, and Emb m It is independent of the number of devices, so the algorithm can adapt to industrial scenarios with different numbers of devices and achieve portability.
[0104] The input of the Transformer module includes E(t) and h m (t-1), where h m (t-1) is the hidden state containing the action and observation history information of device m. The Transformer module processes the matrix E(t) in parallel through the multi-head self-attention mechanism, learns the edge relationship between entities in the input matrix, and obtains the improved graph representation matrix E m (t), and h m (t), where the matrix E mThe vertex vector of (t) fuses the information of adjacent related vertices. The output layer is passed through h m (t) and E m (t) Information generates Q value Q m (τ m ,a m (t)), used to evaluate the quality of the action currently taken by the device, τ m Action-observation history information for device m.
[0105] Secondly, build a Mixing network deployed on the edge server. The core architecture of the Mixing network consists of an Embedder layer, a Transformer module, and an output layer. The output layer consists of two deep neural networks, namely the parameter generation network and the inference network.
[0106] The input of the Embedder layer of the Mixing network is the observation of all devices, defined as s(t). The encoded s(t) is sent to the Transformer module for feature learning, and finally combined with the parameter generation network to generate neuron weights w1 and w2. The final inference network generates a global Q value through neuron weights w1 and w2 and the Q values of all devices. tot (τ, a(t)), the optimization goal of the Agent network is to maximize its own Q value, and the optimization goal of the Mixing network is to minimize Q tot The mean square error between (τ,a(t)) and the target Q value, the Mixing network is constrained Ensure the consistency of the two optimization goals and learn the optimal network resource allocation strategy.
[0107] Finally, a feedback recovery mechanism is designed to deal with the ACK feedback loss caused by imperfect channels, and the training data loss of the neural network is avoided through the recovery mechanism. The feedback recovery mechanism design includes the following steps:
[0108] (1) The channel is modeled as an imperfect channel device with an ε data loss probability, and information about the interaction between the device and the industrial wireless network is collected, including observations, device actions, and rewards, including feedback loss and reward loss information to construct a training dataset;
[0109] (2) The interactions are constructed into a data frame in pandas format and missing data prediction is performed using the deep learning-based DataWig model;
[0110] (3) The ack signal and device action are classified and encoded using one-hot vectors, and feature extraction is performed using the Embedder layer. The remaining data is vector feature concatenated using a normalized neural network.
[0111] (4) Use the classification cross entropy loss function and regression mean square error loss function to jointly train the classification targets (such as ack signals and device actions) and continuous numerical targets (such as other observations, reward values, etc.);
[0112] (5) The output prediction values of the trained neural network model are used to replace the missing feedback in the original data, thereby restoring the complete empirical data to cope with the challenges of imperfect channels to model training.
[0113] like Figure 3 As shown, a joint task offloading and multi-channel access algorithm mainly includes the following steps:
[0114] 1) Initialize the parameters of the algorithm’s neural network model and Transformer module;
[0115] 2) Build a graph structure observation based on the D2D communication range between devices. The features of other devices within the observation range of device m are added to the observation of the device, and the features of devices outside the observation range are filled with 0;
[0116] 3) The device interacts with the industrial wireless network environment according to the graph structure observation output action and obtains rewards. If the feedback signal of the environment interaction is lost due to the interference of the imperfect channel, it is recovered by using the feedback recovery mechanism;
[0117] 4) Collect the complete observations, states, actions, and rewards of industrial equipment as experience trajectories and update the algorithm’s neural network;
[0118] 5) Repeat iterative steps 1)-4) until the number of simulation rounds is reached to complete the training. After the training is completed, each industrial device can perform adaptive task offloading and channel access decisions in a distributed manner to achieve a reasonable allocation of transmission and computing resources in the industrial network. At the same time, the algorithm can adapt to the time-varying number of devices and can cope with feedback loss caused by imperfect channels.
[0119] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for allocating industrial wireless network transmission and computing resources under imperfect channels. It is characterized in that the steps include: S1: Establishing an edge-to-edge collaborative computing architecture for D2D-assisted learning in industrial wireless networks with imperfect channels; S2: Considering the different real-time and computing resource requirements of heterogeneous tasks, we combine multi-channel access and task offloading to obtain a distributed partially observable Markov decision process with a graph structure. S3: Based on a multi-agent deep reinforcement learning framework that integrates Transformer and ACK feedback recovery mechanism, a migratable joint task offloading and multi-channel access algorithm is designed to cope with imperfect channels based on the edge-end collaborative computing architecture, and a computing resource allocation method is obtained. Industrial equipment makes task offloading and multi-channel access decisions based on its own observations to minimize the total delay of task processing.
2. According to the method for allocating industrial wireless network transmission and computing resources under imperfect channels described in claim 1, It is characterized in that The edge-end collaborative computing architecture described in step S1 includes an industrial base station, an edge server, and M industrial devices. The edge server corresponds to the industrial base station. The industrial devices are connected to the edge server through a wireless network. The industrial devices communicate with each other through D2D and generate heterogeneous tasks. The heterogeneous tasks generated by industrial device m at time slot t are d m,n , c m,n , They represent the task size, the required computing delay and the maximum tolerable delay respectively. Represents task n m (t) is a collection of 3. According to the method for allocating industrial wireless network transmission and computing resources under imperfect channels described in claim 2, It is characterized in that The method for obtaining the distributed partially observable Markov decision process of the graph structure in step S2 is: The observable Markov decision process includes graph structure observation, action and reward function. Each industrial device is set as an entity with z features. The observation of industrial device m includes its own features and m-1 features of other entities. The feature information of other entities is obtained through D2D communication. At time slot t, the entity i, i∈M observed by industrial equipment m has its characteristics defined as follows: ent m,i (t)=[ack i (t),d i,n (t),c i,n (t),h i,n (t)]; Among them, ack i (t) indicates the base station confirmation feedback received by entity i, ack i (t)={null,-1,0,1},ack i (t) = null means ack feedback is lost, ack i (t) = -1 indicates that the task transmission fails, ack i (t) = 0 means the task is offloaded locally, ack i (t) = 1 means the task is successfully completed, d i,n (t), c i,n (t), h i,n (t) represents the task n generated by entity i m (t) is the task size, required computing resources and remaining lifetime, n represents task n m (t) index; At time slot t, the graph observation space o of industrial equipment m m (t) is an m*z matrix, expressed as: o m (t)=[ent m,1 (t),ent m,2 (t),...,ent m,i (t)]; The overall observation space of the system can be expressed as: o(t)={o 1 (t),o 2 (t),...,o M (t)}; Among them, ent m,i (t) is the physical feature observed by the industrial equipment, o M (t) is the graph structure observation, and the channel selection action a of industrial equipment m at time slot t m (t) is defined as follows: a m (t)={0,1,...,K}; when a m When (t) = 0, it means that industrial equipment m chooses to offload tasks locally. m When (t) = k, it means that industrial equipment m selects channel k for task offloading; k∈K; The reward function is designed by comprehensively considering the task processing delay optimization and task overdue penalty.
4. According to the method for allocating industrial wireless network transmission and computing resources under imperfect channels described in claim 3, It is characterized in that The method for designing a reward function by comprehensively considering task processing delay optimization and task overdue penalty is: In order to minimize the processing delay of tasks, industrial equipment needs to offload tasks to edge servers for calculation. Therefore, the delay reward d for industrial equipment m at time slot t is m (t) is expressed as: in, For task n m (t) local processing delay at industrial device m, Represents task n m (t) the processing delay offloaded by industrial device m to the edge server; In order to reduce the overdue rate of tasks, the overdue penalty p of industrial equipment m at time slot t is m (t) is expressed as in, For task n m (t) queuing delay in the buffer of industrial equipment m; In summary, at time slot t, the compound reward function r of industrial equipment m is m (t) can be expressed as: r m (t)=d m (t)-α·p m (t); Among them, α is the weight parameter; When the feedback of industrial equipment m is lost, the compound reward function r m (t) is set to null.
5. The industrial wireless network transmission and computing resource allocation method under an imperfect channel according to any one of claims 1 to 4, It is characterized in that The method of designing a joint task offloading and multi-channel access algorithm that can be migrated in the case of imperfect channels described in step S3 is: S31: Build an Agent network deployed on industrial equipment; S32: Build a Mixing network deployed on edge servers; S33: Design a feedback recovery mechanism to cope with ACK feedback loss caused by imperfect channels; S34: A joint task offloading and multi-channel access algorithm that can be transferred to cope with imperfect channels is obtained. During the training period, each industrial device observes the output action according to the graph structure and obtains rewards. The industrial wireless network determines the value of the output action based on the rewards and performs network updates.
6. The industrial wireless network transmission and computing resource allocation method under imperfect channels according to claim 5, It is characterized in that The method of building an Agent network deployed on industrial equipment in step S31 is: The core architecture of the Agent network consists of the Embedding layer, Transformer module, and output layer connected in sequence; The Embedder layer first creates a matrix Emb with a shape of z*h m Observation of graph structure m (t) is encoded, h represents The embedding dimension of the Embedder layer, z represents the matrix Emb m dimension, the final encoded matrix E(t) can be obtained by the formula E(t)=o m (t)Emb m Calculated; o m The same features in (t) are in the matrix Emb m share the same weights, and the matrix Emb m It has nothing to do with the number of industrial equipment; The input of the Transformer module includes the matrix E(t) and h m (t-1), where h m (t-1) is the hidden state containing the action and observation history information of industrial equipment m. The Transformer module processes the matrix E(t) and h in parallel through the multi-head self-attention mechanism. m (t-1), learn the edge relationships between entities in the input matrix and obtain the improved graph representation matrix E m (t) and h m (t), where the matrix E m The vertex vector of (t) fuses the information of adjacent related vertices; the output layer passes through h m (t) and E m (t) Information generates Q value Q m (τ m ,a m (t)), used to evaluate the quality of the current actions taken by industrial equipment, τ m Action-observation history information for industrial equipment.
7. The industrial wireless network transmission and computing resource allocation method under imperfect channels according to claim 6, It is characterized in that The method of step S32 for building a Mixing network deployed on the edge server is: The core architecture of the Mixing network consists of the Embedder layer, Transformer module and output layer connected in sequence. The output layer consists of two deep neural networks, namely the parameter generation network and the inference network. The input of the Embedder layer of the Mixing network is the observation s(t) of all industrial equipment. The encoded observation s(t) is sent to the Transformer module for feature learning, and finally combined with the parameter generation network to generate the neuron weight w 1 With w 2 ; The inference network is constructed by neuron weights w 1 With w 2 and the Q values of all industrial devices to generate a global Q tot (τ, a(t)), the optimization goal of the Agent network is to maximize its own Q value, and the optimization goal of the Mixing network is to minimize Q tot The mean square error between (τ,a(t)) and the target Q value, the Mixing network is constrained Ensure the consistency of the two optimization goals and learn the optimal network resource allocation strategy.
8. The industrial wireless network transmission and computing resource allocation method under imperfect channels according to claim 6 or 7, It is characterized in that Step S33: A feedback recovery mechanism is designed to cope with ACK feedback loss caused by an imperfect channel, including: S331: Model the channel in the edge-to-edge collaborative computing architecture as an imperfect channel industrial device with data loss probability, collect information about the interaction between the industrial device and the industrial wireless network, including observations, industrial device actions and rewards, including feedback loss and reward loss information to build a training data set; S332: Construct the interaction into a data frame in pandas format and use the deep learning-based DataWig model to predict missing data; S333: The ack signal and industrial equipment action in the edge server are classified and encoded using one-hot vectors, and feature extraction is performed using the Embedder layer. The remaining data is vector feature concatenated through a normalized neural network; S334: Use the classification cross entropy loss function and the regression mean square error loss function to jointly train the classification target and the continuous numerical target respectively; S335: Use the trained neural network model output prediction value to replace the missing feedback in the original data to restore the complete empirical data to cope with the challenges of imperfect channels to model training.
9. The industrial wireless network transmission and computing resource allocation method under imperfect channels according to claim 8, It is characterized in that The classification target in step S334 includes the ack signal and the industrial equipment action, and the continuous numerical target includes the remaining observation values and the reward value.
10. The industrial wireless network transmission and computing resource allocation method under imperfect channels according to claim 9, It is characterized in that The steps of the joint task offloading and multi-channel access algorithm for coping with imperfect channels in step S34 are as follows: S341: Initialize the parameters of the neural network model and Transformer module of the algorithm; S342: constructing a graph structure observation according to the D2D communication range between the industrial devices, the features of other industrial devices within the observation range of the industrial device m are added to the observation of the industrial device m, and the features of the industrial devices outside the observation range are filled with 0; S343: The industrial equipment interacts with the industrial wireless network environment according to the graph structure observation output action and obtains rewards. If the feedback signal of the environment interaction is lost due to the interference of the imperfect channel, it is recovered by using the feedback recovery mechanism; S344: Collect the complete observations, states, actions, and rewards of industrial equipment as experience trajectories and update the algorithm’s neural network; S345: Repeat iterating S341-S344 until the number of simulation rounds is reached to complete the training. After the training is completed, each industrial device can perform adaptive task offloading and channel access decisions in a distributed manner to achieve reasonable allocation of transmission and computing resources in the industrial network; The steps of the feedback recovery mechanism in step S344 are: S3441: Model the channel as an imperfect channel device with ε data loss probability, collect information about the interaction between industrial devices and industrial wireless networks, including observations, device actions, and rewards, including feedback loss and reward loss information to construct a training data set; S3442: Construct the interactions into a pandas data frame and use the deep learning-based DataWig model to predict missing data; S3443: The ack signal and device action in the edge server are classified and encoded using one-hot vectors, and feature extraction is performed using the Embedder layer. The remaining data is vector feature concatenated through a normalized neural network; S3444: Use the classification cross entropy loss function and the regression mean square error loss function to jointly train the ack signal and the device action and other observation values and reward values respectively; S3445: Use the trained neural network model output prediction value to replace the missing feedback in the original data to restore the complete empirical data to cope with the challenges of imperfect channels to model training.
Citation Information
Patent Citations
Industrial wireless network resource allocation method based on multi-agent deep reinforcement learning
CN113543156A
Multi-target joint optimization task unloading strategy based on deep reinforcement learning in Internet of Vehicles
CN116321298A
Combined allocation method for communication resources of Internet of Vehicles
CN117915403A
Joint optimization system and method for computation offloading and resource allocation in multi-constraint-edge environment
WO2024065903A1
Cited By
Industrial wireless network resource scheduling method based on multi-weight quality evaluation model
CN119316944A
