Intelligent factory-oriented edge-end cooperative computing task unloading method and system

Abstract

The invention relates to the technical field of industrial Internet of Things and edge computing, and discloses an edge-end cooperative computing task unloading method and system for an intelligent factory, and the method comprises the steps: building a task unloading decision mechanism of a multi-device multi-edge server, distributing a computing node for each task, determining task delay, and constructing a node speciality matrix; establishing an optimization model; formalizing an optimization problem of the optimization model into a Markov decision process; solving a Markov decision process by using a deep reinforcement learning algorithm, and realizing intelligent workshop environment interaction so as to store generated empirical data into an empirical cache region; synchronously updating network parameters of the deep reinforcement learning algorithm and a node speciality matrix corresponding to the edge server by using the data pairs of the experience cache region; and outputting a strategy network and a node speciality matrix until the deep reinforcement learning algorithm converges. And an optimal task allocation decision is dynamically generated, so that the task completion deviation is effectively reduced.

Description

Technical Field

[0001] This invention relates to the field of industrial Internet of Things and edge computing technology, and more specifically, to a method and system for offloading edge-to-edge collaborative computing tasks for smart factories. Background Technology

[0002] In modern smart factories, in order to meet the stringent scheduling requirements of Just-in-Time production, mobile edge computing (MEC) technology is commonly used to offload computationally intensive tasks generated by industrial equipment (such as AI quality inspection and digital twin simulation) to nearby edge servers for processing, thereby reducing latency.

[0003] Existing technologies suffer from two main bottlenecks. First, the granularity of task modeling: Traditional task offloading models typically treat computational tasks as indivisible "black boxes," focusing only on total data volume and computational load while ignoring the internal logical structure of the task. This results in overly coarse task splitting and poor load balancing. Second, resource homogenization: Traditional MEC technologies often treat all edge servers as an undifferentiated, homogeneous pool of computing resources, failing to allocate suitable servers based on task characteristics (e.g., AI inference tasks require servers with high GPU computing power), leading to inefficient resource matching.

[0004] There are currently no effective solutions to the problems in the relevant technologies. Summary of the Invention

[0005] To address the problems in related technologies, this invention proposes an edge-to-edge collaborative computing task offloading method and system for smart factories, in order to overcome the aforementioned technical problems existing in the existing related technologies.

[0006] Therefore, the specific technical solution adopted by the present invention is as follows:

[0007] According to one aspect of the present invention, an edge-to-edge collaborative computing task offloading method for smart factories is provided, comprising:

[0008] Establish a task offloading decision mechanism for multiple devices and multiple edge servers within the smart factory, and construct a node expertise matrix; based on the task offloading decision mechanism, allocate computing nodes for each task and determine the task latency;

[0009] An optimization model is established with the goal of optimizing task delay; the optimization problem of the optimization model is formalized as a Markov decision process.

[0010] Using deep reinforcement learning algorithms, we solve Markov decision processes and realize intelligent workshop environment interaction to store the generated experience data in an experience cache.

[0011] Using data pairs from the experience cache, the network parameters of the deep reinforcement learning algorithm and the node expertise matrix corresponding to the edge server are updated synchronously.

[0012] The process of interacting with the smart factory environment and updating the deep reinforcement learning algorithm is repeated until the deep reinforcement learning algorithm converges, and the policy network and node expertise matrix are output to form the final model of the optimal task unloading decision for the smart factory; the task unloading is completed through the final model.

[0013] Furthermore, establishing a task offloading decision-making mechanism for multiple devices and multiple edge servers within a smart factory includes:

[0014] Configure the equipment for smart factory industrial IoT coverage and deploy edge servers to provide computing power; both the equipment and the edge servers are used as computing nodes.

[0015] At the beginning of each time interval, acquire the macro task generated by the device, and the macro task includes an internal fractal structure;

[0016] For each macro task, a decision is made on whether to perform fractal processing. If fractal processing is selected, each task in the internal fractal structure of the macro task is assigned a computation node independently; if no fractal processing is selected, a computation node is assigned to the macro task.

[0017] The calculation of macro tasks includes the overall calculation of the macro task, as well as the calculation in the order of micro tasks, sub-tasks, and main tasks within the macro task.

[0018] Furthermore, constructing the node expertise matrix includes:

[0019] Use the logarithmic form of heterogeneous computing requirements to determine the feature vector of the task;

[0020] Determine the expertise vector for each edge server, and construct the node expertise matrix corresponding to the edge server based on the expertise vector.

[0021] Furthermore, determining task delays includes:

[0022] Obtain the CPU and GPU computational costs of the macro task, as well as the CPU and GPU computational capabilities of the computing nodes. By comparing the value of CPU computational cost divided by the CPU computational capability of the computing node with the value of GPU computational cost divided by the GPU computational capability, select the largest value as the computational latency of the task.

[0023] The task upload rate is calculated based on the bandwidth during communication, the transmission power when uploading data, the wireless channel gain between the device and the edge server, and the white Gaussian noise power; the data transmission delay of the task is obtained by dividing the amount of data in the task by the task upload rate.

[0024] When a task is assigned to a compute node, the queue waiting delay is calculated based on the task transmission completion time and the task start computation time on the compute node.

[0025] The final completion time of a task is obtained by adding the time when the task starts computing on the computing node to the computing delay; the total delay of a task is obtained by subtracting the final completion time of the task from the time slot.

[0026] When a task is not fractal, it is a macro task; when a task is fractal, it is a micro task, a subtask, or a main task.

[0027] Furthermore, the optimization objective of the optimization model is:

[0028] ;

[0029] The constraints of the optimization model are:

[0030] ;

[0031] ;

[0032] ;

[0033] ;

[0034] ;

[0035] In the formula, Total task delay Indicates the maximum delay period for the task. Let n be the set of devices, where n is the device index.

[0036] For macro tasks It must be assigned to and only assigned to a compute node; For macro tasks fractal tasks The compute node that must be assigned to and only assigned to; The total delay in completing a macro task must not exceed the deadline; For fractal decision variables, they are binary; Decisions for computing nodes are binary;

[0037] Let be the set of devices, and be the computing node. Total task delay For whether or not to make a fractal decision, Decisions are made for the computation nodes of macro tasks. This is for the decision of computation nodes for the fractal task corresponding to the macro task.

[0038] Furthermore, formalizing the optimization problem of the optimization model into a Markov decision process includes:

[0039] The task unloading process is established as a Markov process based on deep reinforcement learning algorithms, including state space, action space and reward function;

[0040] The state space is a vector composed of time slots, local computing queues, transmission queues, server computing queue status, and task attributes.

[0041] The action is an unloading decision vector composed of all terminal devices;

[0042] The reward function follows a Gaussian distribution.

[0043] Further developments and improvements to deep reinforcement learning algorithms include:

[0044] Before the policy network of the deep reinforcement learning algorithm outputs the action probability, the similarity between the feature vector of the task and the expertise vector of each edge server is calculated.

[0045] The similarity is injected as a bias term into the output layer of the policy network and summed with the logarithm of the agent's policy network output to obtain the final decision score, so that the decision can be based on the current state while taking into account the expertise learned from long-term experience.

[0046] The embedding layer is used to map fractal actions to conditional vectors, which serve as additional inputs for computation nodes to select action branches;

[0047] The task offloading decision is broken down into a structured, multi-stage process.

[0048] Furthermore, realizing intelligent workshop environment interaction includes:

[0049] The state vector, which contains the state of all queues and the node expertise matrix corresponding to the edge server at the current moment, is input into the policy network of the deep reinforcement learning algorithm, and the corresponding structured action vector is output.

[0050] Each device terminal executes tasks based on the final allocation decision and receives immediate rewards from the environment based on just-in-time targets;

[0051] The current environmental state, the next environmental state, the structured action vector, and the immediate reward are stored as a four-tuple of experience samples in the experience cache.

[0052] Furthermore, using data pairs from the experience cache, the network parameters of the deep reinforcement learning algorithm and the corresponding node expertise matrix of the edge server are synchronously updated, including:

[0053] Batch sample data is randomly drawn from the experience buffer, and the network parameters of the policy network and value network in the deep reinforcement learning algorithm are updated by calculating the loss function and using the gradient descent optimization algorithm.

[0054] Using the same batch of sample data, the node expertise matrix corresponding to the edge server is updated according to the Heb-like learning rule;

[0055] If the immediate reward obtained from the extracted batch sample data is higher than the current moving average, then the specialty vector of the server selected for the batch sample is updated based on the feature vector of the corresponding batch sample:

[0056] ;

[0057] In the formula, For learning rate, Forgetting factor, For indicator functions, For edge servers Specialization Vector For instant rewards, It is a moving average. These are the eigenvectors.

[0058] According to another aspect of the present invention, an edge-to-edge collaborative computing task offloading system for smart factories is also provided, comprising:

[0059] The node expertise matrix and task delay determination module is used to establish a task offloading decision mechanism for multiple devices and multiple edge servers in a smart factory, and to construct a node expertise matrix; based on the task offloading decision mechanism, computing nodes are allocated to each task, and task delays are determined.

[0060] The module for optimizing temperature calculation and empirical data acquisition is used to establish an optimization model with the goal of optimizing task delay; the optimization problem of the optimization model is formalized into a Markov decision process; deep reinforcement learning algorithm is used to solve the Markov decision process and realize intelligent workshop environment interaction, so as to store the generated empirical data into the empirical cache area.

[0061] The network parameter and node expertise matrix update module is used to synchronously update the network parameters of the deep reinforcement learning algorithm and the node expertise matrix corresponding to the edge server using data pairs from the experience cache.

[0062] The task unloading decision acquisition module is used to repeatedly perform the intelligent workshop environment interaction process and the deep reinforcement learning algorithm update process until the deep reinforcement learning algorithm converges, and outputs the policy network and node expertise matrix to form the final model of the optimal task unloading decision for the intelligent factory; the task unloading is completed through the final model.

[0063] The beneficial effects of this invention are as follows:

[0064] 1. Construct a fractal task model, finely modeling each computational task as a dependent structure that can be fractally divided into core computation (main task), related data processing (sub-task), and edge awareness (micro-task). This innovation solves the granularity problem of task modeling and achieves more refined load balancing by revealing the dependencies between tasks.

[0065] 2. Construct a node expertise model, establishing a dynamically learning and evolving task node matching matrix for each edge server. Servers continuously strengthen their similarity to different task characteristics through historical experience in processing tasks. This innovation solves the problem of resource homogenization, de-averaging computing nodes and enabling servers to self-organize and evolve from a passive, homogenized resource pool into specialized nodes, achieving intelligent matching of computing tasks and resources.

[0066] 3. This invention designs an unloading strategy based on deep reinforcement learning. This strategy can synergistically utilize the two models mentioned above to dynamically generate optimal task allocation decisions, thereby effectively reducing task completion deviations and improving the timeliness of tasks and the overall operating efficiency of the smart factory system. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 This is a flowchart of an edge-to-edge collaborative computing task offloading method for smart factories according to an embodiment of the present invention;

[0069] Figure 2 This is a block diagram of an edge-to-edge collaborative computing task offloading system for smart factories according to an embodiment of the present invention.

[0070] Figure 3 This is one of the model training result diagrams according to an embodiment of the present invention;

[0071] Figure 4 This is the second diagram showing the model training results according to an embodiment of the present invention. Detailed Implementation

[0072] To further illustrate the various embodiments, the present invention provides accompanying drawings, which are part of the disclosure of the present invention. These drawings are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these drawings, those skilled in the art should be able to understand other possible implementation methods and the advantages of the present invention. The components in the drawings are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0073] According to embodiments of the present invention, a method and system for offloading edge-to-edge collaborative computing tasks for smart factories are provided.

[0074] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments, such as... Figure 1 As shown, according to an embodiment of the present invention, an edge-to-edge collaborative computing task offloading method for smart factories is provided, comprising:

[0075] S1. Establish a task offloading decision mechanism for multiple devices and multiple edge servers within the smart factory, and construct a node expertise matrix; based on the task offloading decision mechanism, allocate computing nodes to each task and determine the task latency.

[0076] S2. Establish an optimization model with the goal of optimizing task delay; formalize the optimization problem of the optimization model into a Markov decision process.

[0077] S3. Using deep reinforcement learning algorithms, solve the Markov decision process and realize intelligent workshop environment interaction to store the generated experience data in the experience cache area.

[0078] S4. Using data pairs from the experience cache, synchronously update the network parameters of the deep reinforcement learning algorithm and the node expertise matrix corresponding to the edge server.

[0079] S5. Repeat the intelligent workshop environment interaction process and the deep reinforcement learning algorithm update process until the deep reinforcement learning algorithm converges, and output the policy network and node expertise matrix to form the final model of the optimal task unloading decision of the intelligent factory; complete the task unloading through the final model.

[0080] In one embodiment, establishing a task offloading decision mechanism for multiple devices and multiple edge servers within a smart factory includes:

[0081] Configure the devices covered by the Industrial Internet of Things (IIoT) in the smart factory and deploy edge servers to provide computing power; both the devices and edge servers are used as computing nodes; at the beginning of each time interval, acquire the macro tasks generated by the devices, and the macro tasks include internal fractal structures; for each macro task, decide whether to perform fractal processing. If fractal processing is selected, each task in the internal fractal structure of the macro task is independently assigned a computing node; if fractal processing is not selected, a computing node is assigned to the macro task; the computation of the macro task includes the overall computation of the macro task, as well as the computation in the order of micro tasks, sub-tasks, and main tasks in the macro task.

[0082] In one embodiment, constructing a node expertise matrix includes:

[0083] Using the logarithmic form of heterogeneous computing requirements, the feature vector of the task is determined; the expertise vector of each edge server is determined, and the node expertise matrix corresponding to the edge server is constructed based on the expertise vector.

[0084] In one embodiment, determining the task delay includes:

[0085] The computational costs of a macro-task (CPU and GPU) are obtained, along with the CPU and GPU computing capabilities of the computing nodes. The maximum value (CPU cost divided by the node's CPU capability and GPU cost divided by the node's GPU capability) is selected as the task's computational latency. The task's upload rate is calculated based on bandwidth, transmission power during data upload, wireless channel gain between the device and the edge server, and Gaussian white noise power. The data transmission latency is obtained by dividing the task's data volume by its upload rate. When a task is assigned to a computing node, the queue waiting latency is calculated based on the task's transmission completion time and the time it begins computation on the node. The final completion time of the task is obtained by adding the task's start time to the computational latency. The total task latency is obtained by subtracting the time slot from the final completion time. When the task is non-fractal, it is a macro-task; when the task is fractal, it is classified as a micro-task, sub-task, or main task.

[0086] In one embodiment, the optimization objective of the optimization model is:

[0087] ;

[0088] The constraints of the optimization model are:

[0089] ;

[0090] ;

[0091] ;

[0092] ;

[0093] ;

[0094] In the formula, Total task delay Indicates the maximum delay period for the task. Let n be the set of devices, where n is the device index, and n can take values ​​of 1, 2, ..., N. For macro tasks Decision-making actions; For macro tasks It must be assigned to and only assigned to a compute node; For macro tasks fractal tasks The compute node that must be assigned to and only assigned to; The total delay in completing a macro task must not exceed the deadline; For fractal decision variables, they are binary; Decisions for computing nodes are binary; Let be the set of devices, and be the computing node. Total task delay For whether or not to make a fractal decision, Decisions are made for the computation nodes of macro tasks. This is for the decision of computation nodes for the fractal task corresponding to the macro task.

[0095] In one embodiment, formalizing the optimization problem of the optimization model as a Markov decision process includes:

[0096] The task unloading process is established as a Markov process based on deep reinforcement learning algorithms, including a state space, an action space, and a reward function. The state space is a vector composed of time slots, local computing queues, transmission queues, server computing queue status, and task attributes. The action is an unloading decision vector composed of all terminal devices. The reward function is a Gaussian function distribution.

[0097] In one embodiment, the construction and improvement of a deep reinforcement learning algorithm includes:

[0098] Before the policy network of the deep reinforcement learning algorithm outputs the action probability, the similarity between the feature vector of the task and the expertise vector of each edge server is calculated. The similarity is injected as a bias term into the output layer of the policy network and summed with the output logarithm of the agent's policy network to obtain the final decision score, so that the decision can be based on the current state while taking into account the expertise learned from long-term experience. The fractal action is mapped to a conditional vector using an embedding layer as an additional input for the computation node to select the action branch. The task offloading decision is decomposed into a structured, multi-stage process.

[0099] In one embodiment, enabling smart workshop environment interaction includes:

[0100] The state vector, which contains the state of all queues and the node expertise matrix corresponding to the edge server at the current moment, is input into the policy network of the deep reinforcement learning algorithm, and the corresponding structured action vector is output. Each device terminal executes the task according to the final allocation decision and obtains the instant reward based on the time-definite goal from the environment. The current environmental state, the next environmental state, the structured action vector, and the instant reward are stored as an experience sample quadruple in the experience cache.

[0101] In one embodiment, using data pairs from the experience cache to synchronously update the network parameters of the deep reinforcement learning algorithm and the node expertise matrix corresponding to the edge server includes:

[0102] Batch sample data is randomly drawn from the experience buffer. The network parameters of the policy network and value network in the deep reinforcement learning algorithm are updated by calculating the loss function and using the gradient descent optimization algorithm. Using the same batch of sample data, the node specialty matrix corresponding to the edge server is updated according to the Heb-like learning rule. If the immediate reward obtained from the drawn batch of sample data is higher than the current moving average, the specialty vector of the server selected in the batch of samples is updated according to the feature vector of the corresponding batch of samples.

[0103] ;

[0104] In the formula, For learning rate, Forgetting factor, For indicator functions, For edge servers Specialization Vector For instant rewards, It is a moving average. These are the eigenvectors.

[0105] like Figure 2 As shown, according to another embodiment of the present invention, an edge-to-edge collaborative computing task offloading system for smart factories is also provided, comprising:

[0106] The node expertise matrix and task delay determination module 1 is used to establish a task offloading decision mechanism for multiple devices and multiple edge servers in a smart factory, and to construct a node expertise matrix; according to the task offloading decision mechanism, it allocates computing nodes to each task and determines the task delay.

[0107] The temperature calculation and experience data acquisition module 2 is used to establish an optimization model with the goal of optimizing task delay; the optimization problem of the optimization model is formalized into a Markov decision process; the Markov decision process is solved using a deep reinforcement learning algorithm, and the intelligent workshop environment interaction is realized so that the generated experience data can be stored in the experience cache area.

[0108] The network parameter and node expertise matrix update module 3 is used to synchronously update the network parameters of the deep reinforcement learning algorithm and the node expertise matrix corresponding to the edge server using data pairs from the experience cache.

[0109] The task unloading decision acquisition module 4 is used to repeatedly perform the intelligent workshop environment interaction process and the deep reinforcement learning algorithm update process until the deep reinforcement learning algorithm converges, and outputs the policy network and node expertise matrix to form the final model of the optimal task unloading decision of the intelligent factory; the task unloading is completed through the final model.

[0110] To facilitate understanding of the above technical solutions of the present invention, the working principle of the present invention in actual process will be described in detail below.

[0111] This invention includes the following steps: Step 1: Establishing a multi-device, multi-server correspondence, including a task offloading model with macroscopic attributes and internal fractal structures. Step 2: Establishing a latency model for handling heterogeneous computing demands under the fractal task structure concept. Step 3: Establishing a problem model with the optimization objective of minimizing the deviation between early arrival and late arrival times. Step 4: Formalizing the problem in Step 3 as a Markov decision process. Step 5: Designing a deep reinforcement learning algorithm to solve the model in Step 4. Step 6: Interacting with the intelligent workshop environment using the algorithm designed in Step 5, storing data experience in an experience cache. Step 7: Using the data in the experience cache to initiate the dual learning and update process, synchronously updating the network parameters of the deep reinforcement learning algorithm and the node expertise matrix of the server. Step 8: Repeating Steps 6 and 7 until the algorithm converges and finds the optimal solution.

[0112] The process consists of five steps: Step 1 is the definition and modeling of devices, servers, and tasks; Step 2 is the action decision and the corresponding physical rules based on Step 1; Step 3 is the definition and constraints of the goal and the optimization direction based on the results of the decision in Step 2; Step 4 is to transform the problem in Step 3 into a language that AI (reinforcement learning) can understand, including states, actions, and rewards; and Step 5 is to design a tool to solve the specific problem constructed by the first four steps. The first four steps construct a detailed model of the problem and transform it into the language of deep reinforcement learning. Step 5 designs an algorithm to understand the language and solve the constructed problem.

[0113] The model constructed in process one is as follows:

[0114] Consider a MEC task offloading scenario in a smart factory, consisting of multiple industrial devices and multiple edge servers that provide additional computing power.

[0115] 1.1 Edge Task Model:

[0116] The collection of devices covered by the smart factory's industrial IoT network is , For device indexing, a collection of edge servers deployed around it to provide computing power is... System time is defined as a time slot set. The devices capable of performing calculations and edge servers are collectively referred to as compute nodes. .

[0117] At the start of each time slot, each device generates a macro task. ,in, Indicates the macroscopic attributes of the task. This indicates its internal fractal structure. ,in, Indicates the amount of data in the task, This indicates the CPU computational load of the task. This indicates the GPU computational cost of the task. Indicates the maximum delay period for the task; Each task Each has its own independent attributes in, Indicates the main task , Subtasks , Microtasks The computational complexity of the task satisfies:

[0118] ;

[0119] For the generated macro-tasks, they can be calculated as a whole, or they can be calculated according to their internal fractal structure in the order of micro, sub, and principal. Taking a product quality inspection scenario as an example, the corresponding task description is as follows:

[0120] Macro task: Responsible for executing a complete end-to-end task. It encompasses the entire process from receiving raw data input to producing the final analysis results. It is executed on a single computing node.

[0121] Microtask: Responsible for front-end data acquisition. Extracting keyframes from industrial video streams.

[0122] Subtask: Responsible for preprocessing the collected data. Perform image preprocessing on the extracted video frames.

[0123] Main task: Responsible for the core and most complex calculations. Perform AI inference on preprocessed images.

[0124] 1.2 Node Expertise Model:

[0125] The logarithmic form of the heterogeneous computing requirements is used to define the feature vector of the task. :

[0126] ;

[0127] Specialization vectors for each server The definition is as follows:

[0128] ;

[0129] The node expertise model maintains an M×2 dimensional matrix of server expertise. :

[0130] ;

[0131] This matrix dynamically and adaptively evolves during the learning process. Among them, The CPU requirements for the task. The GPU computing requirements for the task. The CPU expertise similarity of the server at the current moment is... This represents the similarity of the server's GPU expertise at the current moment.

[0132] 1.3 Task Unloading Decision:

[0133] For each macro task generated by the device, the decision is divided into two parts: whether to use fractals and the allocation of computing nodes.

[0134] The decision-making process is as follows: For a specific macro-task, first decide whether to perform fractal analysis; if fractal analysis is chosen, then proceed with the structure described. Each task in Specify a compute node independently If non-fractal is selected, a computation node is specified for the overall macro-task. During device-side calculations, When the computation is offloaded to any edge server, .

[0135] The model constructed in process two is as follows:

[0136] For each time slot set The core of the unloading decision is for each macro task or its internal fractal structure All tasks Allocate computing nodes.

[0137] Based on the allocation decision, the subsequent processing flow of the task component is as follows: Local execution path: If the task is allocated to a local device, it directly enters the local computing queue on that device. Offload execution path: If the task is allocated to an edge server, it first enters the transmission queue on that device, and after the data transmission is completed, it enters the computing queue of the target server.

[0138] 2.1 Calculation of Delay Model:

[0139] Computational latency depends on the task's computational demands on the CPU and GPU, as well as the corresponding computing capabilities of the compute nodes. Each compute node... The computing power is defined as , representing the processing power of its CPU and GPU, respectively.

[0140] (1) Task is not fractal:

[0141] Macro tasks generated by the time device At the node The computational task has a computational latency of As shown in the following formula:

[0142] ;

[0143] in, The CPU computational cost of the task. This represents the GPU computational cost of the task.

[0144] (2) Task fractals:

[0145] Macro tasks generated by the time device Each internal fractal task At the node Its computational delay is As shown in the following formula:

[0146] ;

[0147] 2.2 Data transmission delay model:

[0148] When a task's computing node selects an edge server, it needs to upload data via a wireless link. The latency generated during the transmission process is mainly determined by the amount of data and the transmission rate, with the specific upload rate determined by Shannon's formula.

[0149] (1) Task is not fractal:

[0150] ;

[0151] (2) Task fractals:

[0152] ;

[0153] in, Bandwidth during communication. The transmission power when uploading data. For the wireless channel gain between the device and the server, The power is Gaussian white noise. Because the network is relatively stable in the smart factory environment, the task... upload speed at any time Constant and unchanging.

[0154] (1) The task is not fractal, and the task upload latency is:

[0155] ;

[0156] in, For macro tasks The amount of data, This refers to the upload speed.

[0157] (2) Task fractal, task upload latency is:

[0158] ;

[0159] in, For the task The amount of data, This refers to the upload speed.

[0160] 2.3 Queue Waiting Delay Model Definition of Devices in the System The completion time of the transmission queue is The completion time of the computation queue is The computation queue of server M completes at [time]. . For binary indicator functions, the computation node of the task. The time is 0.

[0161] (1) Task is not fractal:

[0162] equipment generated macro tasks Assigned to the target computing node The task transmission completion time is:

[0163] ;

[0164] The task begins computation on the compute node at the following time:

[0165] .

[0166] (2) Task fractals:

[0167] equipment generated macro tasks fractal tasks Assigned to the target computing node The task transmission queue completion time is:

[0168] ;

[0169] The node computation begins at:

[0170] ;

[0171] in, Microtasks represent the completion time of the preceding tasks of the current task. .

[0172] 2.4 Total Task Delay Model:

[0173] (1) Task is not fractal:

[0174] Macro Tasks The final completion moment of completion for:

[0175] ;

[0176] The total delay is:

[0177] .

[0178] (2) Task fractals:

[0179] Macro Tasks fractal tasks completion time :

[0180] ;

[0181] The total delay is:

[0182] .

[0183] The model constructed in process three is as follows:

[0184] The delay for each task in each time slot is The optimization objective is:

[0185] ;

[0186] :

[0187] ;

[0188] ;

[0189] ;

[0190] ;

[0191] .

[0192] in, For macro tasks It must be assigned to one and only one compute node. For macro tasks fractal tasks It must be assigned to one and only one compute node. The total delay in completing a macro task must not exceed the deadline. The fractal decision variables are binary. The decision-making process for computing nodes is binary.

[0193] The task offloading strategy for deep reinforcement learning constructed in process four is as follows:

[0194] The task unloading process is modeled as a standard Markov process, the basis of deep reinforcement learning. Markov decision processes primarily consist of three key elements: state, action, and reward. The time slots are then used to... The observed state space is defined as a vector composed of time slots, local computing queues, transmission queues, server computing queue status, and task attributes, as shown below:

[0195] ;

[0196] in, For the current time interval, For the device end Calculate the queue completion time. For the device end Transmission queue completion time, For edge servers Calculate the queue completion time. The macro task that arrives at the right time The set, This is a matrix of server specializations.

[0197] Action space Defined as the offloading decision vector composed of all terminal devices, specifically expressed as:

[0198] ;

[0199] in, The decision of whether to break down the task. Assignment strategies to specific compute nodes:

[0200] ;

[0201] in, For the computation nodes of macro tasks, The main task computation node. Calculate nodes for subtasks. This is a microtask computing node.

[0202] The reward function adopts a Gaussian distribution form:

[0203] ;

[0204] in, This represents the maximum reward that can be obtained if the task is completed on time. The variance control reward of the Gaussian function is sensitive to delay bias. These are the weighting coefficients of the linear penalty term. This represents the total deviation of the macro task. This is the maximum extension period.

[0205] The improved deep reinforcement learning algorithm for near-end policy optimization constructed in step five is as follows:

[0206] 5.1 Expertise-biased decision-making:

[0207] Introducing a server expertise matrix This biases the decision-making process to achieve intelligent task routing. Before the policy network outputs the action probability, the task type vector is first calculated. With each server Specialty vector The similarity between the agents is used as a bias term and injected into the output layer of the policy network. This bias is then summed with the logarithm of the agent's policy network output to obtain the final decision score. This allows decisions to be based on the current state while also taking into account the expertise learned from long-term experience. The specific expressions for the calculation formula and sampling strategy are as follows:

[0208] (1) Task is not fractal:

[0209] ;

[0210] ;

[0211] (2) Task fractals:

[0212] ;

[0213] ;

[0214] in, To perform summation and normalization, all computation nodes are traversed. This is used as a weight to adjust the degree of influence of the expertise bias. Specifically, if you choose to calculate it directly on the device side, the bias term defaults to 0. The bias computation vector for macro tasks is a server-specific feature. For the server's expertise vector, To share the backbone network.

[0215] 5.2 Conditional Action Dependency:

[0216] Whether or not a fractal decision is made for a macro-task directly affects the selection of computing nodes. Establishing this dependency helps in finding the optimal strategy. Through a conditional network structure, the output of the computing nodes depends on the selection of fractal actions. Specifically, an embedding layer is used to map fractal actions into conditional vectors, which serve as additional inputs for the computing nodes to select action branches.

[0217] ;

[0218] ;

[0219] in, To share the backbone network, the parameters are: Extract features from the original state; For fractal decision heads, the parameters are: Output fractal decision; This is the decision head for the computation node, with the following parameters: Output the decision on selecting computing nodes; As an embedding layer, it maps discrete decision symbols into continuous vectors. For fractal action strategies, Choose a strategy for compute nodes. Let be the state space vector at the current moment.

[0220] 5.3 Action Stratified Sampling:

[0221] Unloading decisions are a hierarchical process, and joint sampling decomposes the decision into a structured, multi-stage process. (Agent Policy Network) For a complete action The output probability follows the chain rule in probability theory; the AI ​​first outputs a probability distribution about the execution structure. After sampling, based on the high-level strategy, low-level decisions are... Select computation nodes and sample a set of nodes. The specific formula is as follows:

[0222] .

[0223] The specific interactions in process six are as follows:

[0224] 1. State Input and Decision Output: The current state vector, containing all queue states and the server specialty matrix. Inputting data into the policy network of a deep reinforcement learning algorithm, the algorithm outputs corresponding structured action vectors. This includes structural decisions for task execution, as well as decisions for allocating computing nodes based on that structure.

[0225] 2. Execution and Feedback: Each terminal device executes the task according to the final allocation decision. Upon completion of the task, it receives an immediate reward from the environment based on the just-in-time target. .

[0226] 3. State transition and storage: The environment state changes accordingly. Update to the next moment's state And the experience sample quadruple generated from this interaction ( Store it in the experience cache area.

[0227] The specific steps for process seven are as follows:

[0228] 1. Randomly sample batches of data from the experience buffer, calculate the loss function, and use the gradient descent optimization algorithm to update the network parameters of the policy network and value network in the deep reinforcement learning algorithm. This improves the agent's decision-making ability.

[0229] 2. Using the same batch of sample data, and based on the Heb-like learning rule, analyze the server expertise matrix. Update. If the extracted data sample yields an immediate reward... Higher than the current moving average Then, based on the feature vector of the sample Servers selected for this sample Specialization Vector Update.

[0230] :

[0231] in, It is a relatively small learning rate, representing the speed at which specialized memories are formed; It's a forgetting factor that ensures expertise evolves over time. It is an indicator function. By optimizing the task matching capability with the server computing nodes, the system can achieve self-organization and long-term evolution.

[0232] The specific steps for process eight are as follows:

[0233] Repeat the agent-environment interaction process described in step six and the learning and updating process described in step seven until the cumulative reward value or system cost of the algorithm tends to stabilize and converge. When the algorithm converges, the obtained policy network and expertise matrix together constitute the final model that can output the optimal task unloading decision for the smart factory.

[0234] In practical applications, to demonstrate the main usage process, consider a simple scenario with one device and two edge servers:

[0235] I. Task Generation and Model Definition:

[0236] A certain time interval in the production of a smart factory Industrial robots A product quality inspection task needs to be performed, which involves capturing images of the product to be inspected using an equipped camera and performing inference using an AI model. The macro-task generated by this process is defined as follows: This is an analysis of a two-second video segment, with the feature vector being... .

[0237] The corresponding fractal task is defined as:

[0238] Microtasks Extract keyframes from the video. The feature vector is... Subtasks Image preprocessing is performed on keyframes. The feature vector is... Main task An AI model is used to perform inference analysis on keyframes. The feature vector is... .

[0239] II. The agent makes decisions:

[0240] 1. State observation:

[0241] State vector This includes the current time, queue status, pending task attributes, and server expertise matrix.

[0242] .

[0243] 2. Decision Output:

[0244] Agent Policy Network State vector Then, a hierarchical decision is made, and the action vector for the current task is output. :

[0245] .

[0246] The process is illustrated using the main task as an example; the same applies to other tasks.

[0247] Offset calculation:

[0248] ;

[0249] ;

[0250] Probability calculation (w=5):

[0251] ;

[0252] ;

[0253] ;

[0254] III. Task Execution and Delay Calculation:

[0255] Based on the above decision Macro tasks use fractal processing, micro tasks The computation is performed on server 2, a subtask. The main task is to unload the computation onto server 1. The calculations are then performed on server 1.

[0256] 1. Micro-tasks:

[0257] ;

[0258] ;

[0259] ;

[0260] ;

[0261] .

[0262] in, The time required for the transmission of microtasks. The time required for computation of the microtask. The time when the microtask transmission is completed. The calculation time is the start time of the microtask. The completion time of the microtask is calculated.

[0263] 2. Subtasks:

[0264] ;

[0265] ;

[0266] ;

[0267] ;

[0268] .

[0269] in, The time required for the transmission of subtasks. The time required to compute the subtask. Transmit the completion time for the subtask. The start time of the subtask is calculated. Calculate the completion time for each subtask.

[0270] 3. Main Task:

[0271] ;

[0272] ;

[0273] ;

[0274] ;

[0275] ;

[0276] .

[0277] in, The time required for the main task's transmission. The time required for the main task's computation. The time when the main task transmission is completed. The start time of the main task calculation. Calculate the completion time for the main task.

[0278] III. Reward Value Calculation and Experience Storage:

[0279] The reward value for this decision is calculated based on the reward function. The completion time is very close to the deadline, and it is a large positive value.

[0280] ;

[0281] The experience sample quadruple generated in this interaction ( Stored in the experience cache. Among them:

[0282] ;

[0283] ;

[0284] .

[0285] IV. Algorithm Network Learning and Updates:

[0286] 1. Policy network update:

[0287] When the number of experience samples in the experience buffer reaches a preset update threshold, the system will periodically initiate a dual learning and update process. This process randomly selects a batch of samples from the experience buffer; in this sample, the agent... Make a decision Obtain a larger reward value A strong positive feedback signal will guide the gradient descent algorithm to adjust the parameters of the policy network. To improve future encounters with Make decisions in similar situations The probability of.

[0288] 2. Node expertise update:

[0289] Server 1:

[0290] ;

[0291] ;

[0292] .

[0293] Server 2:

[0294] ;

[0295] .

[0296] V. Continuous interaction and convergence:

[0297] By repeatedly performing the above process, the algorithm's reward value and bias index converge. At this point, the agent has learned the optimal fractal and node selection strategies in the output policy network under various load conditions and different types of tasks. The training results are shown in the figure below. Figure 3 and Figure 4 As shown, the model converges stably, and the total deviation value stabilizes around a small value. Episode represents rounds / rounds; Reward represents the reward; TotalDeviations(s) represents the total deviation / total deviation amount.

[0298] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for offloading edge-to-edge collaborative computing tasks in smart factories, characterized in that, include: Establish a task offloading decision mechanism for multiple devices and multiple edge servers within the smart factory, and construct a node expertise matrix; Based on the task unloading decision mechanism, a computing node is allocated to each task, and the task delay is determined; An optimization model is established with the goal of optimizing task delay; the optimization problem of the optimization model is formalized as a Markov decision process. Using deep reinforcement learning algorithms, we solve Markov decision processes and realize intelligent workshop environment interaction to store the generated experience data in an experience cache. Using data pairs from the experience cache, the network parameters of the deep reinforcement learning algorithm and the node expertise matrix corresponding to the edge server are updated synchronously. The process of interacting with the smart factory environment and updating the deep reinforcement learning algorithm is repeated until the deep reinforcement learning algorithm converges, and the policy network and node expertise matrix are output to form the final model of the optimal task unloading decision for the smart factory; the actual task unloading is completed through the solution results of the final model.

2. The edge-to-edge collaborative computing task offloading method for smart factories according to claim 1, characterized in that, The established task offloading decision mechanism for multiple devices and multiple edge servers within a smart factory includes: Configure the equipment for smart factory industrial IoT coverage and deploy edge servers to provide computing power; both the equipment and the edge servers are used as computing nodes. At the beginning of each time interval, acquire the macro task generated by the device, and the macro task includes an internal fractal structure; For each macro task, a decision is made on whether to perform fractal processing. If fractal processing is selected, each task in the internal fractal structure of the macro task is assigned a computation node independently; if no fractal processing is selected, a computation node is assigned to the macro task. The calculation of macro tasks includes the overall calculation of the macro task, as well as the calculation in the order of micro tasks, sub-tasks, and main tasks within the macro task.

3. The edge-to-edge collaborative computing task offloading method for smart factories according to claim 1, characterized in that, The constructed node expertise matrix includes: Use the logarithmic form of heterogeneous computing requirements to determine the feature vector of the task; Determine the expertise vector for each edge server, and construct the node expertise matrix corresponding to the edge server based on the expertise vector.

4. The edge-to-edge collaborative computing task offloading method for smart factories according to claim 1, characterized in that, The determination of task delay includes: Obtain the CPU and GPU computational costs of the macro task, as well as the CPU and GPU computational capabilities of the computing nodes. By comparing the value of CPU computational cost divided by the CPU computational capability of the computing node with the value of GPU computational cost divided by the GPU computational capability, select the largest value as the computational latency of the task. The task upload rate is calculated based on the bandwidth during communication, the transmission power when uploading data, the wireless channel gain between the device and the edge server, and the white Gaussian noise power; the data transmission delay of the task is obtained by dividing the amount of data in the task by the task upload rate. When a task is assigned to a compute node, the queue waiting delay is calculated based on the task transmission completion time and the task start computation time on the compute node. The final completion time of a task is obtained by adding the time when the task starts computing on the computing node to the computing delay; the total delay of a task is obtained by subtracting the final completion time of the task from the time slot. When a task is not fractal, it is a macro task; when a task is fractal, it is a micro task, a subtask, or a main task.

5. The edge-to-edge collaborative computing task offloading method for smart factories according to claim 1, characterized in that, The optimization objective of the optimization model is: ； The constraints of the optimization model are: ； ； ； ； ； In the formula, Total task delay Indicates the maximum delay period for the task. Let n be the set of devices, where n is the device index. For macro tasks It must be assigned to and only assigned to a compute node; For macro tasks fractal tasks The compute node that must be assigned to and only assigned to; The total delay in completing a macro task must not exceed the deadline; For fractal decision variables, they are binary; Decisions for computing nodes are binary; Let be the set of devices, and be the computing node. Total task delay For whether or not to make a fractal decision, Decisions are made for the computation nodes of macro tasks. This is for the decision of computation nodes for the fractal task corresponding to the macro task.

6. The edge-to-edge collaborative computing task offloading method for smart factories according to claim 1, characterized in that, The process of formalizing the optimization problem of the optimization model into a Markov decision process includes: The task unloading process is established as a Markov process based on deep reinforcement learning algorithms, including state space, action space and reward function; The state space is a vector composed of time slots, local computing queues, transmission queues, server computing queue status, and task attributes. The action is an unloading decision vector composed of all terminal devices; The reward function is a Gaussian function distribution.

7. The edge-to-edge collaborative computing task offloading method for smart factories according to claim 1, characterized in that, The construction and improvement of the deep reinforcement learning algorithm include: Before the policy network of the deep reinforcement learning algorithm outputs the action probability, the similarity between the feature vector of the task and the expertise vector of each edge server is calculated. The similarity is injected as a bias term into the output layer of the policy network and summed with the logarithm of the agent's policy network output to obtain the final decision score, so that the decision can be based on the current state while taking into account the expertise learned from long-term experience. The embedding layer is used to map fractal actions to conditional vectors, which serve as additional inputs for computation nodes to select action branches; The task offloading decision is broken down into a structured, multi-stage process.

8. The edge-to-edge collaborative computing task offloading method for smart factories according to claim 1, characterized in that, The implementation of intelligent workshop environment interaction includes: The state vector, which contains the state of all queues and the node expertise matrix corresponding to the edge server at the current moment, is input into the policy network of the deep reinforcement learning algorithm, and the corresponding structured action vector is output. Each device terminal executes tasks based on the final allocation decision and receives immediate rewards from the environment based on just-in-time targets; The current environmental state, the next environmental state, the structured action vector, and the immediate reward are stored as a four-tuple of experience samples in the experience cache.

9. A method for offloading edge-to-edge collaborative computing tasks in a smart factory according to claim 1, characterized in that, The data pairs used in the experience cache are used to synchronously update the network parameters of the deep reinforcement learning algorithm and the node expertise matrix corresponding to the edge server, including: Batch sample data is randomly drawn from the experience buffer, and the network parameters of the policy network and value network in the deep reinforcement learning algorithm are updated by calculating the loss function and using the gradient descent optimization algorithm. Using the same batch of sample data, the node specialty matrix corresponding to the edge server is updated according to the Heb-like learning rule; If the immediate reward obtained from the extracted batch sample data is higher than the current moving average, then the specialty vector of the server selected for the batch sample is updated based on the feature vector of the corresponding batch sample: ； In the formula, For learning rate, Forgetting factor, For indicator functions, For edge servers Specialization Vector For instant rewards, It is a moving average. These are the eigenvectors.

10. An edge-to-edge collaborative computing task offloading system for smart factories, used to implement the edge-to-edge collaborative computing task offloading method for smart factories as described in any one of claims 1-9, characterized in that, include: The node expertise matrix and task delay determination module is used to establish a task offloading decision mechanism for multiple devices and multiple edge servers in a smart factory, and to construct a node expertise matrix. Based on the task unloading decision mechanism, a computing node is allocated to each task, and the task delay is determined; The module for optimizing temperature calculation and empirical data acquisition is used to build an optimization model with the goal of optimizing task delay; the optimization problem of the optimization model is formalized into a Markov decision process; Using deep reinforcement learning algorithms, we solve Markov decision processes and realize intelligent workshop environment interaction to store the generated experience data in an experience cache. The network parameter and node expertise matrix update module is used to synchronously update the network parameters of the deep reinforcement learning algorithm and the node expertise matrix corresponding to the edge server using data pairs from the experience cache. The task unloading decision acquisition module is used to repeatedly perform the intelligent workshop environment interaction process and the deep reinforcement learning algorithm update process until the deep reinforcement learning algorithm converges, and outputs the policy network and node expertise matrix to form the final model of the optimal task unloading decision for the intelligent factory; the task unloading is completed through the final model.

Images