A heuristic algorithm-based method for offloading augmented reality tasks in an edge-to-edge collaborative environment

By subdividing AR tasks into subtasks and optimizing the unloading strategy using Bayesian networks and heuristic algorithms, the problem of minimizing AR task latency in edge-device collaborative environments is solved, enabling low-latency operation of AR applications and improving user experience.

CN115718648BActive Publication Date: 2026-11-10YUNNAN NORMAL UNIV
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202211356049.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-11-01
Publication Date
2026-11-10
Estimated Expiration
2042-11-01

AI Technical Summary

Technical Problem

In an edge-to-device collaborative environment, existing technologies have failed to effectively address how to reasonably offload augmented reality (AR) tasks to different edge servers to minimize the overall task latency, taking into account the heterogeneity of edge computing platforms and the temporal relationships of AR sub-tasks.

Method used

The AR task is divided into fine-grained subtasks. Bayesian networks are used to evaluate the feature dependencies of edge servers. Particle swarm optimization and genetic algorithms are used to optimize the unloading strategy, which is then abstracted into a traveling salesman problem to minimize the total task latency.

Benefits of technology

It effectively reduces the latency of AR tasks under conditions of limited mobile terminal resources, improves the immersive experience of AR applications, and reduces user dizziness.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115718648B_ABST
    Figure CN115718648B_ABST
Patent Text Reader

Abstract

This invention relates to an augmented reality task offloading method based on heuristic algorithms in an edge-device collaborative environment. The invention first divides the overall AR task into several fine-grained subtasks and determines the size of each AR subtask; it then identifies the underlying environmental characteristics of the edge server that affect the execution latency of the AR subtasks; it collects "feature-latency" data, constructs a batch normalization (BN) algorithm, and predicts the execution latency of the AR subtasks on the edge server; it calculates the execution latency of the AR subtasks on the mobile terminal and the total latency of offloading to the edge server; it uses the point sorting algorithm (PSO) to encode each AR task offloading scheme, and after using a general algorithm (GA) to accelerate convergence, it provides the scheme with the shortest total latency as the final AR task offloading scheme. This invention can design a reasonable AR task offloading strategy, effectively reduce the total execution latency of AR tasks, and return the user's interaction results with the scene more quickly.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to an augmented reality task offloading method based on heuristic algorithms in an edge-device collaborative environment, belonging to the field of task offloading technology under edge-device collaboration. Background Technology

[0002] With the widespread adoption of mobile devices, mobile terminals running on different operating systems have introduced support for Augmented Reality (AR) technology, such as Google's ARCore and Apple's ARKit. Due to their ability to deeply integrate digital and physical world information, AR applications developed based on mobile terminals will inevitably change the current way humans interact with computers.

[0003] Ensuring low latency is a crucial quality of service (QoS) metric for AR applications running on mobile devices, as it reduces user dizziness and enhances the immersive experience. However, from a technical implementation perspective, a complete AR application task consists of five fine-grained subtasks executed sequentially: video capture, location tracking, target mapping, target detection, and result presentation. Each subtask requires significant CPU computing power, network bandwidth, memory capacity, and other resources. Therefore, maintaining low latency for mobile devices with limited hardware resources presents a significant challenge.

[0004] In traditional cloud-centric service models, these resource-intensive AR subtasks can be offloaded to cloud servers, leveraging their powerful clusters for task and data processing and analysis. However, long-distance transmission can increase latency and the risk of user privacy breaches. In recent years, the rise of Edge Computing (EC) has provided a compromise to these challenges. EC can offload some or all of the AR subtasks generated by mobile terminals to multiple edge servers with abundant CPU and memory resources. This lightweight edge-to-device collaborative service model reduces energy consumption and latency during task transmission between the edge and cloud, alleviates pressure on central cloud servers, and ensures parallel execution of tasks on edge servers.

[0005] However, in an edge-to-device collaborative environment, it is challenging to reasonably offload multiple AR subtasks to different edge servers to minimize the overall latency of the AR task. This is specifically reflected in the following two aspects.

[0006] (1) When designing the offloading strategy for each AR subtask, it is necessary to first evaluate and predict its execution latency on each edge server, and then plan the offloading scheme based on the prediction results. However, due to the heterogeneity of the edge computing platform, it has differentiated underlying environment characteristics (including: software characteristics, hardware characteristics, configuration characteristics, and runtime environment characteristics). There are uncertain nonlinear dependencies between these characteristics, as well as between the characteristics and the execution latency of the AR subtask, which poses a challenge to the prediction of execution latency.

[0007] (2) Since the execution of each AR subtask has a temporal relationship, the implementation process of the overall AR task is as follows: Figure 1 As shown in (a), this process can be abstracted into a... Figure 1 (b) shows a weighted directed graph. The nodes are... to Let each node represent an AR subtask, and the different directed edges and their weights between nodes represent different subtask unloading schemes and their corresponding delays. Therefore, designing a scheme that optimizes the total AR task execution delay can be viewed as finding a path between different nodes, ultimately forming a shortest cycle. This is the Traveling Salesman Problem (TSP) in graph theory, and it is NP-hard.

[0008] In existing technologies, task offloading methods in edge-device collaborative environments can be broadly classified into three categories: those based on mathematical optimization, those based on machine learning, and those based on engineering control.

[0009] The task offloading method based on mathematical optimization establishes an optimization model involving indicators such as energy consumption, latency, network bandwidth, and load balancing. Then, it uses integer programming, heuristic algorithms, and game theory to find the optimal solution from among many candidate options. The paper "Resource Allocation and Task Offloading Scheme Based on Game Theory in Cloud-Edge Collaborative Systems," published in the Journal of System Simulation (2022), comprehensively considers latency, energy consumption, and computing resources to construct a utility maximization problem in cloud-edge collaborative systems. It establishes the problem model using game theory-based resource allocation and task offloading schemes and obtains the optimal solution using convex optimization conditions and a game theory-based distributed task offloading algorithm.

[0010] Machine learning-based task offloading methods primarily involve collecting "task feature-offloading target" data and then using supervised learning, unsupervised learning, and reinforcement learning to build a task offloading model. This allows the model to predict and provide solutions when new tasks arrive. The paper "An Edge-Cloud Collaborative Resource Allocation Algorithm Based on Deep Reinforcement Learning," published in Computer Science, 2022, proposes a deep reinforcement learning-based task offloading strategy to reduce the total latency of task offloading across multiple terminals. Simulation results demonstrate that the algorithm exhibits good convergence.

[0011] Task offloading based on engineering control involves real-time monitoring and analysis of task implementation, and timely adjustment of the offloading strategy when violations of task-defined performance metrics such as latency occur. The Chinese invention patent (CN113590232A), published in 2021, titled "A Task Offloading Method for Relay Edge Networks Based on Digital Twins," establishes a task offloading strategy model for relay edge networks. The current model and the optimal task offloading strategy model are forwarded from the edge server to the relay node, and then forwarded by the relay node to the user terminal communicating with it. The user terminal and relay node perform corresponding task offloading according to the optimal task offloading strategy model.

[0012] The aforementioned public knowledge indicates that most current offloading schemes primarily target single, non-continuous tasks. Therefore, this invention addresses the characteristic that AR tasks can be divided into fine-grained subtasks, and that the execution of each subtask is temporally sequential, proposing an edge-to-edge collaborative AR task offloading method. First, a Bayesian Network (BN) is constructed using hill climbing and Maximum Likelihood Estimation (MLE) to assess and quantify the uncertain dependencies between multiple underlying environmental features of the edge server, thereby achieving accurate prediction of the execution latency of each AR subtask on different edge servers. Then, the offloading of the overall AR task is abstracted into solving a Time-of-Stake (TSP) problem. Multiple offloading schemes are encoded into particles using the Particle Swarm Algorithm (PSO), and crossover and mutation operations in a Genetic Algorithm (GA) are used to accelerate particle convergence, ultimately obtaining an AR task offloading scheme that minimizes the total latency.

[0013] Task offloading has been widely applied in edge-to-edge collaboration in recent years. However, there are currently no reports on abstracting AR task offloading into solving the TSP problem and using Bayesian networks and heuristic algorithms to obtain a solution that minimizes the execution latency of AR tasks. Summary of the Invention

[0014] To address the aforementioned issues, this invention provides an augmented reality task offloading method based on heuristic algorithms in an edge-device collaborative environment. This method also aims to alleviate the limitation that the latency of resource-intensive augmented reality (AR) tasks running on mobile terminals cannot be guaranteed due to limited hardware resources. This invention provides an AR task offloading scheme that minimizes the total latency.

[0015] The technical solution of this invention is: an augmented reality task offloading method based on heuristic algorithms in an edge-to-edge collaborative environment, the specific steps of which are as follows:

[0016] Step 1: Divide the overall AR task into several fine-grained subtasks and determine the size of each AR subtask;

[0017] Step 2: Determine the underlying environmental characteristics of the edge server that affect the execution latency of AR subtasks;

[0018] Step 3: Collect "feature-latency" data, construct BN, and predict the execution latency of AR subtasks on the edge server;

[0019] Step 4: Calculate the execution latency of the AR subtask on the mobile terminal and the total latency of offloading to the edge server;

[0020] Step 5: Encode the unloading schemes for each AR task using the PSO algorithm, and after the GA acceleration scheme converges, give the scheme with the shortest total latency as the final unloading scheme for the AR task.

[0021] As a further aspect of the present invention, in Step 1, an overall AR task is divided into the following five fine-grained sub-tasks: (1) Video acquisition, responsible for editing the original video from the mobile terminal into multiple frames of images; (2) Position tracking, responsible for determining the user's position in the real environment; (3) Target mapping, responsible for converting the position tracking results into three-dimensional geometric coordinates; (4) Target detection, responsible for using label-free or label-based computer vision techniques to detect virtual objects; (5) Result presentation, responsible for displaying the enhanced information through the terminal; the above tasks are respectively used as ~ The size of each task is represented as... .

[0022] As a further aspect of the present invention, in Step 2, the underlying environment characteristics of the edge server include software characteristics, hardware characteristics, configuration characteristics, and runtime environment characteristics; by adjusting different combinations of the above four characteristics and the differentiated values ​​of each characteristic, an edge server with multiple sets of characteristic configurations is obtained.

[0023] As a further aspect of the present invention, Step 3 includes the following steps:

[0024] Step 3.1: Dataset preparation: Run different AR subtasks on the edge server with the fixed feature configuration in Step 2, and collect the latency data of the AR subtasks under the current configuration;

[0025] Step 3.2: Utilizing the dataset Building BN: Using the collected dataset, construct the corresponding BN through structure building and parameter calculation;

[0026] Step 3.3: Use the constructed BN to predict the latency of AR subtasks running on different edge servers.

[0027] As a further aspect of the present invention, Step 3.1 includes:

[0028] Assuming that the selection was made in Step 2 If fi is a feature that affects the runtime latency of an AR subtask, then the fi-th feature is represented as: The configuration of the edge server is represented as The runtime latency of the corresponding subtask is represented as... AR subtasks In configuration Repeated run Subsequently, the collected "feature-latency" dataset was represented as .

[0029] As a further aspect of the present invention, Step 3.2 includes the following steps:

[0030] Step 3.2.1: Define the variables related to BN and give the constraints in the BN construction process;

[0031] (1) Define the structure of BN as follows ,in, This represents a node in BN, which includes the characteristics of edge servers and time-delay nodes, i.e.: ;and Let $\mathbf{N}$ be the set of directed edges in BN, where $\mathbf{N}$ nodes are $\mathbf{N}$. The directed edges between them are represented as ,therefore ; The parameters represent BN, which contain the conditional probability table of all nodes in BN;

[0032] (2) Define nodes The set of parent nodes is And it has a total of Various combination forms; using This indicates when the value of the node is... And its parent node is the first The probability of such combinations; using To represent probability values ,and It is the BN parameter One of the possible values ​​is: ;

[0033] The following three constraints are given to optimize the BN structure;

[0034] Constraint 1: Initial structure: Create whitelists and blacklists respectively for the relationships between directed edges in BN. The former records the edges that must be included in BN, and the latter records mutually independent feature nodes; determine the initial structure based on the whitelists and blacklists of BN.

[0035] Constraint 2: Parameter Calculation: Parameter calculation is for the purpose of calculating... The value when the node If there is no parent node, then , The value is determined by The prior probability determines it; otherwise, when the node The parent node is the first When there is a combination, then The value of is calculated from the conditional probability, that is... ,at the same time ;

[0036] Constraint 3: Structure Construction: The underlying environmental characteristics of the edge server affect the execution latency of AR subtasks, but the execution latency cannot, in turn, affect the corresponding features; this relationship is represented in Batch Normalization (BN) as a relationship between feature nodes. To the delayed node A directed edge Furthermore, the direction of the directed edges cannot be reversed. In addition, all directed edges in BN are not allowed to form cycles.

[0037] Step 3.2.2: Parameter calculation and structure construction of BN;

[0038] Assuming the collected If all the "feature-delay" data satisfy the condition of independent and identically distributed, then we first define a node. The characteristic function is shown in formula (1), and is used with express and The number of data instances at that time;

[0039] (1)

[0040] According to the definition of formula (1), The calculation method is as follows: Therefore, according to the maximum likelihood estimation method, for nodes... Different values For example, its calculation method is shown in formula (2);

[0041] (2)

[0042] in, To meet the conditions The total number of data entries at that time;

[0043] After calculating the conditional probability of each node in BN according to formula (2), all the conditional probability tables constitute the parameters of BN. ,Right now: ;

[0044] Next, based on the parameters of BN and the current structure The Bayesian Information Criterion (BIC) is used to calculate the value of a given feature-latency dataset. When, the degree of fit between the structure and parameters of BN; the calculation formula is shown in formula (3);

[0045] (3)

[0046] in, Let be the number of independent parameters in BN; therefore, Considered as a penalty for the complexity of the BN structure, the more independent nodes there are in the BN, the smaller the BIC score corresponding to the structure, proving that the complexity of the BN is lower; in particular, when all nodes in the BN are independent, the BN is a Naive Bayes network.

[0047] When a given node and its parent node When this occurs, the BIC score corresponding to this pair of parent and child nodes is calculated as shown in formula (4);

[0048] (4)

[0049] Formula (5) is derived from formulas (3) and (4), which means that the BIC score of the entire BN structure is obtained by adding the BIC scores of multiple substructures;

[0050] (5)

[0051] Finally, the hill climbing method is used to search for the optimal BN structure. Taking the constraint (1) given in Step 3.2.1 as the starting point of the search, multiple candidate BN structures are obtained by adding, subtracting, and flipping edges on the current BN structure.

[0052] Assume that the node at this time The parent node is the previous one. Become Then, at this point, the candidate structure... The BIC score is calculated using formula (6);

[0053] (6)

[0055] in, and These represent the BN structure after the hill-climbing search and before the search, respectively.

[0056] Given a fixed threshold ,if The value is less than If so, then the hill-climbing method converges, and the corresponding structure This is the optimal BN structure that is ultimately constructed. ;

[0057] Step 3.2.3: Use the constructed BN to predict the execution latency of AR subtasks on the edge server;

[0058] Complete structure and parameters based on Batch Normalization And multiple underlying environmental characteristics of edge servers can predict AR subtasks. In the Execution latency on edge servers , is the delay node in BN. The result corresponding to the maximum conditional probability value, that is:

[0059] .

[0060] As a further aspect of the present invention, Step 4 includes the following steps:

[0061] Step 4.1: First, define the symbols: Assume there are n edge servers, where the k-th server is represented as... ( When the task On mobile terminals When the above is executed, its corresponding latency is expressed as ; and when it is on the edge server When executed, the corresponding latency is mobile terminal The CPU clock speed is ,Task The CPU cycles required to run on it are ;

[0062] therefore, On mobile terminals The execution delay calculation method is shown in formula (7);

[0063] (7)

[0064] And if Offloaded to edge server Execution time should include the task's transmission delay and the latency in time. The execution latency, i.e.: ;in, The value is obtained from the BN prediction constructed in Step 3; while for Let's first assume a mobile terminal. and edge servers The upload and download bandwidth between them is consistent and without fluctuation; its value is... Therefore, when the task The size is When the transmission delay is calculated, the method is as shown in formula (8);

[0065] (8)

[0066] therefore, Unload to edge server The total time delay is calculated as shown in formula (9);

[0067] (9).

[0068] As a further aspect of the present invention, Step 5 includes the following steps:

[0069] Step 5.1 First, the unloading strategy encoding utilizes the idea of ​​the PSO algorithm, with the aim of encoding multiple unloading strategies of the AR overall task into multiple corresponding particles;

[0070] When AR Total Task During the first When iterating the wheel unloading strategy, the corresponding particle encoding method is defined as follows:

[0071]

[0072] in, Indicates task A feasible uninstallation solution, This indicates a subtask. The execution platform is At that time, the corresponding delay , It includes both AR terminals and edge server platforms. Indicates task The convergence speed, Indicate uninstallation scheme During the update, the number of AR subtasks to be updated on the execution platform is: ;

[0073] Step 5.2, after considering AR subtasks On mobile terminals Execution delay and offloading to edge servers Total latency Then, the fitness function is designed as shown in formula (10);

[0074] (10)

[0075] in, ,and , , From the perspective of AR subtasks, the latency of each execution scheme was measured, and From the perspective of the overall task, the total latency of the five AR subtasks was measured. The smaller the value, the more reasonable the corresponding task unloading scheme.

[0076] Step 5.3 Next, design the iteration and update of the PSO algorithm;

[0077] The update of the AR task offloading scheme based on the PSO algorithm involves three factors: intrinsic factors, local factors, and global factors. These correspond to the update method of the strategy, the subtask, and the overall strategy, respectively. Uninstallation strategy updates and overall tasks Update the uninstallation strategy; During the wheel unloading strategy update process, the subtasks are made... Corresponding The unloading strategy with the minimum value is the locally optimal strategy, denoted as . ; and thus the overall task Corresponding The unloading strategy with the smallest value is the globally optimal strategy, denoted as . ;

[0078] Since the update of the unloading strategy is a discrete rather than continuous process, a particle update scheme as shown in formula (11) was designed using the GA algorithm.

[0079] (11)

[0080] in

[0081] (12)

[0082] Indicates when the policy update factor Greater than random number At that time, for the uninstallation scheme Perform single-point mutation operation ,Right now: In the plan In the middle, change any subtask execution platform ;

[0083] (13)

[0084] This indicates that the roulette wheel algorithm is used when the subtask policy update factor... Greater than random number Then, make the subtask The corresponding task unloading scheme with the shortest latency and the modified uninstallation scheme Perform single-point crossover operation ,Right now: Indicates the uninstallation scheme In the middle, sub-tasks The execution platform was replaced with The corresponding execution platform ;

[0085] (14)

[0086] This indicates that the roulette wheel algorithm is used when the total task update factor is... Greater than random number At that time, in the plan and the globally optimal uninstallation solution In the middle, select any subtask Perform single-point crossover operation ;Right now: Indicates the uninstallation scheme In the middle, sub-tasks The execution platform was replaced with Execution platform for corresponding subtasks ;

[0087] Will , , Repeat execution in sequence After that, we got the first In the round strategy iteration process Several candidate solutions; subsequently, in this Of the three schemes, the one that maximizes the fitness function shown in formula (6) is selected as the task unloading scheme for the next round of strategy iteration. ;

[0088] Repeat Steps 5.2 and 5.3 until the fitness function is achieved. Convergence to threshold Then, the corresponding uninstallation scheme This is the final AR task unloading solution. After deploying this solution to the corresponding mobile terminal or edge service platform, it can ensure the shortest latency during the task unloading process.

[0089] The beneficial effects of this invention are:

[0090] 1. In an environment where edge servers and mobile terminals work together, this invention alleviates the limitation that the latency of resource-intensive augmented reality (AR) tasks running on mobile terminals cannot be guaranteed due to the limited hardware resources of mobile terminals. Therefore, it proposes a strategy to offload AR tasks to edge servers that are physically far away but have sufficient hardware resources.

[0091] 2. This invention designs a reasonable AR task offloading strategy, offloading the task from the resource-poor mobile terminal to the edge computing platform with abundant hardware resources and closer proximity, thereby effectively reducing the total execution latency of AR tasks and returning the user's interaction results with the scene faster. Attached Figure Description

[0092] Figure 1 An AR task unloading model is defined; where (a) is the AR task execution flow; and (b) is the AR task execution abstraction flow.

[0093] Figure 2 This is a flowchart of the present invention;

[0094] Figure 3 This is an initial schematic diagram of the BN of the present invention; wherein (a) is the structure and parameters of the initial corresponding BN; and (b) is the set of data instances used to construct the BN.

[0095] Figure 4 This is a schematic diagram of the updated BN structure according to the present invention, wherein (a) is the structure and parameters of the updated BN; and (b) is the set of data instances used to update the BN. Detailed Implementation

[0096] Example 1: As Figures 1-4 As shown, an augmented reality task offloading method based on a heuristic algorithm in an edge-device collaborative environment is proposed. This invention aims to comprehensively select a scheme that minimizes the total AR task latency after considering the execution latency of multiple AR subtasks on the mobile terminal and the total latency of offloading to the edge server (including task transmission latency and execution latency on the edge server). The invention includes the following specific steps:

[0097] Step 1: Divide the overall AR task into 5 fine-grained subtasks and determine the size of each AR subtask.

[0098] An overall AR task is precisely divided into 5 fine-grained sub-tasks, namely: =Video capture, =Location tracking, =Target mapping, =Object detection, =Results presentation. And the size of each task is as follows: , , , , ,

[0099] Step 2: Determine the underlying environmental characteristics of the edge server that affect the execution latency of AR subtasks.

[0100] The underlying environment feature set is determined as: F = {CPU microarchitecture type, number of virtual CPUs, LLC capacity, CPU scheduling algorithm};

[0101] Step 3: Collect "feature-latency" data, construct BN, and predict the execution latency of AR subtasks on the edge server.

[0102] Step 3.1: Dataset preparation.

[0103] By modifying the configuration of several underlying environment features in Step 2 and running the AR subtask on them, the corresponding execution latency data can be obtained. Among them, subtasks The “feature-latency” data is shown in Table 2, which contains 6 data instances.

[0104] Table 2 lists the subtasks. The "feature-task data" set D

[0105]

[0106] Step 3.2: Utilizing the dataset Building BN: Using the collected dataset, construct the corresponding BN through structure building and parameter calculation;

[0107] Step 3.3: Use the constructed BN to predict the latency of AR subtasks running on different edge servers.

[0108] The steps in Step 3.2 are as follows:

[0109] Step 3.2.1: Define the variables related to BN and give the constraints in the BN construction process.

[0110] First, define the structure of BN as follows: .in, These are feature nodes, including the features and performance shown in Table 2. That is: And edge set It is used to connect nodes. ,node The directed edges between them are represented as ,therefore ;, The parameters represent BN, which contain the conditional probability table of all nodes in BN;

[0111] Define nodes The set of parent nodes is And it has a total of Combination form; using This indicates when the value of the node is... And its parent node is the first The probability of such combinations; using To represent probability values ,and It is the BN parameter One of the possible values ​​is: ;

[0112] Then, the following constraints are given to optimize the BN structure.

[0113] Constraint 1: Initial Structure: Define the whitelist and blacklist for edges in Batch Normalization (BN). Based on prior knowledge of the hardware on the edge server, it can be determined that the characteristic of "CPU microarchitecture type" definitely affects "LLC capacity". Therefore, the whitelist for edges in BN is... However, the necessary independent relationship between the features involved in Step 2 cannot be determined at present, therefore the blacklist for BN cannot be determined.

[0114] Constraint 2: Parameter Calculation. Using the six data points shown in Table 2, calculate the initial parameters of BN. First, encode the relevant feature nodes with serial numbers: #1 represents the "CPU microarchitecture type" node, #2 represents the "number of virtual CPUs" node, and so on. Assume the parameters to be calculated at this point are... By examining the data in Table 2, we can see that this parameter represents the probability value when the "LLC capacity" of node 3 is the second case and the number of parent nodes is the first combination. Therefore, according to formulas (1) and (2), we can obtain the following results:

[0115] .

[0116] Similarly, the parameters of each node can be calculated in this way.

[0117] Constraint 3: Structure Construction: The underlying environmental characteristics of the edge server affect the execution latency of AR subtasks, but the execution latency cannot, in turn, affect the corresponding features; this relationship is represented in Batch Normalization (BN) as a relationship between feature nodes. To the delayed node A directed edge Furthermore, its direction cannot be reversed. In addition, all directed edges in BN are not allowed to form cycles. Specifically, by combining the calculated parameters of each node and the whitelist of edges in BN, an initial structure and parameters of BN as shown in Figure (3) can be determined. The number of each node is given in parentheses after the node name.

[0118] Step 3.2.2: Parameter calculation and structure construction of BN.

[0119] Assuming the collected If all the "feature-delay" data satisfy the condition of independent and identically distributed, then we first define a node. The characteristic function is shown in formula (1), and is used with express and The number of data instances at that time;

[0120] (1)

[0121] According to the definition of formula (1), The calculation method is as follows: Therefore, according to the maximum likelihood estimation method, for nodes... Different values For example, its calculation method is shown in formula (2);

[0122] (2)

[0123] in, To meet the conditions The total number of data entries at that time;

[0124] After calculating the conditional probability of each node in BN according to formula (2), all the conditional probability tables constitute the parameters of BN. ,Right now: ;

[0125] Next, based on the parameters of BN and the current structure The Bayesian Information Criterion (BIC) is used to calculate the value of a given feature-latency dataset. When, the degree of fit between the structure and parameters of BN; the calculation formula is shown in formula (3);

[0126] (3)

[0127] in, Let be the number of independent parameters in BN; therefore, Considered as a penalty for the complexity of the BN structure, the more independent nodes there are in the BN, the smaller the BIC score corresponding to the structure, proving that the complexity of the BN is lower; in particular, when all nodes in the BN are independent, the BN is a Naive Bayes network.

[0128] When a given node and its parent node When this occurs, the BIC score corresponding to this pair of parent and child nodes is calculated as shown in formula (4);

[0129] (4)

[0130] Specifically, in this embodiment, the parameters in the initial BN are calculated according to formulas (1)-(4). and each substructure , as well as The BIC scoring is implemented as follows.

[0131]

[0132]

[0133]

[0134] Formula (5) is derived from formulas (3) and (4), which means that the BIC score of the entire BN structure is obtained by adding the BIC scores of multiple substructures;

[0135] (5)

[0136] According to formula (5), the BIC score of the entire BN structure can be calculated as follows:

[0137]

[0138] Finally, the hill climbing method is used to search for the optimal BN structure. Taking the constraint (1) given in Step 3.2.1 as the starting point of the search, multiple candidate BN structures are obtained by adding, subtracting, and flipping edges on the current BN structure.

[0139] Specifically, the current structure is iterated using the hill-climbing method. The operations performed are: deleting the directed edge between node 2 ("Number of Virtual CPUs") and node 4 ("CPU Scheduling Algorithm"), and recalculating the BIC score corresponding to the iteratively updated BN structure. The updated BN is: The corresponding structure and parameters are shown in Figure (4).

[0140] Assume that the node at this time The parent node is the previous one. Become Then, at this point, the candidate structure... The BIC score is calculated using formula (6);

[0141] (6)

[0142] in, and These represent the BN structure after the hill-climbing search and before the search, respectively.

[0143] Furthermore, specifically, The corresponding BIC score calculation details are as follows:

[0144]

[0145] Set a fixed threshold .because Therefore, the structure of BN converges, and the corresponding structure This is the optimal BN structure that is ultimately constructed. ;

[0146] Step 3.2.3: Use the constructed BN to predict the execution latency of AR subtasks on the edge server.

[0147] Based on the BN constructed in Step 3.2.2, predict when the features on a given edge server are... At that time, AR subtask The execution latency on it, namely: Similarly, all other subtasks Execution latency on edge servers can be predicted using similar methods.

[0148] Step 4: Calculate the execution latency of the AR subtask on the mobile terminal and the total latency of offloading to the edge server.

[0149] First, calculate each subtask on the mobile terminal. Execution latency Assume the mobile terminal's clock speed is currently... And each subtask requires 100 CPU clock cycles to run, that is The sizes of each subtask are as follows: Therefore, according to formula (7);

[0150] (7)

[0151] Computable subtasks The execution latencies on mobile terminals are as follows:

[0152]

[0153]

[0154]

[0155]

[0156]

[0157] Next, the total latency of the computing task if it is offloaded to the edge server.

[0158] Table 2 shows six different edge server configurations. Based on Step 3, the AR subtasks can be predicted first. The execution latency on these six edge servers is shown in Table 3. That is, the corresponding subtask The prediction results.

[0159] Table 3 shows the execution latency of AR subtasks on the edge server.

[0160]

[0161] if Offloaded to edge server Execution time should include the task's transmission delay and the latency in time. The execution latency, i.e.: ;in, The value is obtained from the BN prediction constructed in Step 3; while for Let's first assume a mobile terminal. and edge servers The upload and download bandwidth between them is consistent and without fluctuation; its value is... Therefore, when the task The size is When the transmission delay is calculated, the method is as shown in formula (8);

[0162] (8)

[0163] therefore, Unload to edge server The total time delay is calculated as shown in formula (9);

[0164] (9).

[0165] Specifically, computational tasks Uninstall to The transmission delay at that time. Assume that the uplink bandwidth and download bandwidth are the same at this time and... According to formula (8), the subtasks can be calculated. Unload to any edge The transmission delays at each time are:

[0166]

[0167]

[0168]

[0169]

[0170]

[0171] Based on the execution latency and transmission latency predicted by BN, the subtask can be determined according to formula (9). If the load is offloaded to an edge server, the corresponding total latency is shown in Table 4.

[0172] Table 4 shows the total unloading time for each subtask.

[0173]

[0174] Step 5: Encode the unloading schemes for each AR task using the PSO algorithm, and after the GA acceleration scheme converges, give the scheme with the shortest total latency as the final unloading scheme for the AR task.

[0175] Step 5.1: First, give an initial population of 3 particles and set the update rate. Based on the subtasks calculated in Step 4, on the mobile terminal... Execution latency, and offloading to edge servers The total delay can be used to set the encoding information of the three particles as follows:

[0176]

[0177]

[0178]

[0179] Step 5.2, after considering AR subtasks On mobile terminals Execution delay and offloading to edge servers Total latency Then, the fitness function is designed as shown in formula (10);

[0180] (10)

[0181] in, ,and , , From the perspective of AR subtasks, the latency of each execution scheme was measured, and From the perspective of the overall task, the total latency of the five AR subtasks was measured. The smaller the value, the more reasonable the corresponding task unloading scheme.

[0182] Specifically, the fitness of the above three particles is calculated according to formula (10), and the results are as follows:

[0183] 63.75

[0184] 29.97

[0185]

[0186] Step 5.3 Next, design the iteration and update of the PSO algorithm;

[0187] Perform the first round of particle iteration and update;

[0188] The update of the AR task offloading scheme based on the PSO algorithm involves three factors: intrinsic factors, local factors, and global factors. These correspond to the update method of the strategy, the subtask, and the overall strategy, respectively. Uninstallation strategy updates and overall tasks Update the uninstallation strategy; During the wheel unloading strategy update process, the subtasks are made... Corresponding The unloading strategy with the minimum value is the locally optimal strategy, denoted as . ; and thus the overall task Corresponding The unloading strategy with the smallest value is the globally optimal strategy, denoted as . ;

[0189] Since the update of the unloading strategy is a discrete rather than continuous process, a particle update scheme as shown in formula (11) was designed using the GA algorithm.

[0190] (11)

[0191] in

[0192] (12)

[0193] Indicates when the policy update factor Greater than random number At that time, for the uninstallation scheme Perform single-point mutation operation ,Right now: In the plan In the middle, change any subtask execution platform ;

[0194] (13)

[0195] Specifically, first, set the strategy update factor. , , However, by using a pseudo-random algorithm, random numbers can be obtained. .

[0196] According to formula (12), since Therefore, a single-point mutation operation is performed on each of the three particles, that is, any subtask from a certain particle is randomly selected. The execution platform has been changed. The updated particle configuration is shown below.

[0197]

[0198]

[0199]

[0200] According to formula (13), since Therefore, the function is not executed, meaning that the single-point crossover operation between the above three particles and the locally optimal particle is not performed.

[0201] Formula (13) represents the algorithm using the roulette wheel betting method, where the subtask strategy update factor is... Greater than random number Then, make the subtask The corresponding task unloading scheme with the shortest latency and the modified uninstallation scheme Perform single-point crossover operation ,Right now: Indicates the uninstallation scheme In the middle, sub-tasks The execution platform was replaced with The corresponding execution platform ;

[0202] (14)

[0203] This indicates that the roulette wheel algorithm is used when the total task update factor is... Greater than random number At that time, in the plan and the globally optimal uninstallation solution In the middle, select any subtask Perform single-point crossover operation ;Right now: Indicates the uninstallation scheme In the middle, sub-tasks The execution platform was replaced with Execution platform for corresponding subtasks ;

[0204] According to formula (14), since Therefore, a single-point crossover operation is performed on the above three particles and the globally optimal particle. According to the definition of the PSO algorithm, the globally optimal particle is the unloading strategy corresponding to the lowest fitness value. In the middle, the globally optimal particle Therefore, we select Subtasks in The corresponding execution platform, i.e. , respectively with particles , Neutron mission The execution platform is replaced, and the latency corresponding to each subtask is determined based on the calculation results of Step 4. After this operation, the updated particles are:

[0205]

[0206]

[0207]

[0208] Assuming the number of iterations Furthermore, the convergence threshold of the PSO-GA algorithm At this point, the entire algorithm iteration is complete, and the fitness of each particle is:

[0209] 7.817

[0210] 29.913

[0211]

[0212] In the above results, the encoding corresponding to minimum fitness is: ,and .therefore, The corresponding strategy is the final AR task unloading strategy, which means that a complete AR task is divided into 5 independent sub-tasks, among which... The operating platform is a mobile terminal. The operating platform is an edge server. , The operating platform is an edge server. , The operating platform is an edge server. , The operating platform is an edge server. This solution is the optimal AR task offloading solution and can guarantee the shortest total latency.

[0213] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A method for offloading augmented reality tasks based on heuristic algorithms in an edge-to-edge collaborative environment, characterized in that: The specific steps of the method are as follows: Step 1: Divide the overall augmented reality task into several fine-grained subtasks and determine the size of each augmented reality subtask; Step 2: Determine the underlying environmental characteristics of the edge server that affect the execution latency of augmented reality subtasks; Step 3: Collect "feature-latency" data, construct a Bayesian network, and predict the execution latency of augmented reality subtasks on the edge server; Step 4: Calculate the execution latency of the augmented reality subtask on the mobile terminal and the total latency of offloading to the edge server; Step 5: Encode each augmented reality task unloading scheme using the particle swarm optimization algorithm, and use the genetic algorithm to accelerate the convergence of the schemes. Then, give the scheme with the shortest total latency as the final unloading scheme for the augmented reality task. In Step 1, an augmented reality task is divided into the following 5 fine-grained subtasks: (1) video acquisition, which is responsible for editing the original video of the mobile terminal into multiple frames of images; (2) position tracking, which is responsible for determining the user's position in the real environment; (3) target mapping, which is responsible for converting the position tracking results into three-dimensional geometric coordinates; (4) target detection, which is responsible for using label-free or label-based computer vision technology to detect virtual objects; and (5) result presentation, which is responsible for displaying the augmented information through the terminal. The above tasks are respectively used ~ The size of each task is represented as... ; In Step 2, the underlying environment characteristics of the edge server include software characteristics, hardware characteristics, configuration characteristics, and runtime environment characteristics. By adjusting different combinations of the above four characteristics and the differentiated values ​​of each characteristic, edge servers with multiple sets of characteristic configurations are obtained. Step 3 includes the following steps: Step 3.1: Dataset preparation: Run different augmented reality subtasks on the edge servers with fixed feature configurations in Step 2, and collect the latency data corresponding to the augmented reality subtasks under the current configuration; Step 3.2: Utilizing the dataset Building a Bayesian network: Using the collected dataset, construct the corresponding Bayesian network through structure building and parameter calculation; Step 3.3: Use the constructed Bayesian network to predict the latency of augmented reality subtasks running on different edge servers.

2. The augmented reality task offloading method based on heuristic algorithms in an edge-to-edge collaborative environment according to claim 1, characterized in that: Step 3.1 includes: Assuming that the selection was made in Step 2 If the feature is used as a feature affecting the runtime latency of augmented reality subtasks, then the feature is... fi Each feature is represented as , The runtime latency of the corresponding subtask is represented as... Augmented reality subtasks , In configuration Repeated run Subsequently, the collected "feature-latency" dataset was represented as .

3. The augmented reality task offloading method based on heuristic algorithms in an edge-to-edge collaborative environment according to claim 1, characterized in that: Step 3.2 includes the following steps: Step 3.2.1: Define the variables related to the Bayesian network and give the constraints in the process of constructing the Bayesian network; (1) Define the structure of a Bayesian network as follows: ,in, This represents a node in a Bayesian network, which includes the characteristics of edge servers and time-delay nodes, i.e.: ;and Let $\mathbf{ ... The directed edges between them are represented as ,therefore ; This represents the parameters of a Bayesian network, which contains a table of conditional probabilities for all nodes in the Bayesian network. (2) Define nodes The set of parent nodes is And it has a total of Various combination forms; using This indicates when the value of the node is... And its parent node is the first The probability of such combinations; using To represent probability values ,and These are Bayesian network parameters. One of the possible values ​​is: ; The following three constraints are given to optimize the Bayesian network structure; Constraint 1: Initial Structure: Create whitelists and blacklists for the relationships between directed edges in the Bayesian network. The former records edges that must be included in the Bayesian network, while the latter records mutually independent feature nodes. Determine the initial structure of the Bayesian network based on its whitelists and blacklists. Constraint 2: Parameter Calculation: Parameter calculation is for the purpose of calculating... The value when the node If there is no parent node, then , The value is determined by The prior probability determines it; otherwise, when the node The parent node is the first When there is a combination, then The value of is calculated from the conditional probability, that is... ,at the same time ; Constraint 3: Structure Construction: The underlying environmental characteristics of the edge server affect the execution latency of augmented reality subtasks, but the execution latency cannot, in turn, affect the corresponding characteristics; this relationship is represented in Bayesian networks as a relationship between feature nodes. To the delayed node A directed edge Furthermore, the direction of the directed edges cannot be reversed. In addition, all directed edges in a Bayesian network are not allowed to form cycles. Step 3.2.2: Parameter calculation and structure construction of Bayesian networks; Assuming the collected If all the "feature-delay" data satisfy the condition of independent and identically distributed, then we first define a node. The characteristic function is shown in formula (1), and is used with express and The number of data instances at that time; (1); According to the definition of formula (1), The calculation method is as follows: Therefore, according to the maximum likelihood estimation method, for nodes... Different values For example, its calculation method is shown in formula (2); (2); in, To meet the conditions The total number of data entries at that time; After calculating the conditional probability of each node in the Bayesian network according to formula (2), all the conditional probability tables constitute the parameters of the Bayesian network. ,Right now: ; Next, based on the parameters of the Bayesian network... and the current structure The Bayesian Information Criterion (BIC) is used to calculate the value of a given "feature-delay" dataset. The degree of fit between the structure and parameters of the Bayesian network is calculated using the formula (3). (3); in, Let be the number of independent parameters in the Bayesian network; therefore, Considered as a penalty for the complexity of the Bayesian network structure, the more independent nodes there are in the Bayesian network, the smaller the BIC score corresponding to the structure, proving that the complexity of the Bayesian network is lower; in particular, when all nodes in the Bayesian network are independent, the Bayesian network is a Naive Bayesian network. When a given node and its parent node When this occurs, the BIC score corresponding to this pair of parent and child nodes is calculated as shown in formula (4); (4); Formula (5) is derived from formulas (3) and (4), which means that the BIC score of the entire Bayesian network structure is obtained by adding the BIC scores of multiple substructures; (5); Finally, the hill-climbing method is used to search for the optimal Bayesian network structure. Starting from the constraint 1 given in Step 3.2.1, multiple candidate Bayesian network structures are obtained by adding, subtracting, and flipping edges on the current Bayesian network structure. Assume that the node at this time The parent node is the previous one. Become Then, at this point, the candidate structure... The BIC score is calculated using formula (6); (6); in, and These represent the Bayesian network structures before and after the hill-climbing search, respectively. Given a fixed threshold ,if The value is less than If so, then the hill-climbing method converges, and the corresponding structure This is the optimal Bayesian network structure that is ultimately constructed. ; Step 3.2.3: Use the constructed Bayesian network to predict the execution latency of augmented reality subtasks on the edge server; Complete structure and parameters based on Bayesian networks As well as multiple underlying environmental characteristics of edge servers, it is possible to predict augmented reality subtasks. In the Execution latency on edge servers For delay nodes in a Bayesian network The result corresponding to the maximum conditional probability value, that is: .

4. The augmented reality task offloading method based on heuristic algorithms in an edge-to-edge collaborative environment according to claim 1, characterized in that: Step 4 includes the following steps: Step 4.1: First, define the symbols: Assume there are n edge servers, where the k-th server is represented as... , When subtasks On mobile terminals When the above is executed, its corresponding latency is expressed as ; and when it is on the edge server When executed, the corresponding latency is mobile terminal The CPU clock speed is Subtask The CPU cycles required to run on it are ; therefore, On mobile terminals The execution delay calculation method is shown in formula (7); (7); And if Offloaded to edge server Execution time should include the task's transmission delay and the latency in time. The execution latency, i.e.: ;in, The value is predicted by the Bayesian network constructed in Step 3; while for Let's first assume a mobile terminal. and edge servers The upload and download bandwidth between them is consistent and without fluctuation; its value is... Therefore, when subtasks The size is When the transmission delay is calculated, the method is as shown in formula (8); (8); therefore, Unload to edge server The total time delay is calculated as shown in formula (9); (9)。 5. The augmented reality task offloading method based on heuristic algorithms in an edge-to-edge collaborative environment according to claim 1, characterized in that: Step 5 includes the following steps: Step 5.1 First, the unloading strategy encoding utilizes the idea of ​​the particle swarm algorithm, with the aim of encoding multiple unloading strategies of the overall augmented reality task into corresponding multiple particles. When Augmented Reality Total Task During the first When iterating the wheel unloading strategy, the corresponding particle encoding method is defined as follows: ; in, Indicates task A feasible uninstallation solution, This indicates a subtask. The execution platform is At that time, the corresponding delay , It includes both augmented reality terminals and edge server platforms. Indicates task The convergence speed, Indicate uninstallation scheme During the update, the number of augmented reality subtasks to be updated on the platform is: ; Step 5.2, considering augmented reality subtasks On mobile terminals Execution latency and offloading to edge servers Total latency Then, the fitness function is designed as shown in formula (10); (10); in, ,and , , From the perspective of augmented reality subtasks, the latency of each execution scheme was measured, and From the perspective of the overall task, the total latency of the five augmented reality subtasks was measured. The smaller the value, the more reasonable the corresponding task unloading scheme. Step 5.3 Next, design the iteration and update of the particle swarm optimization algorithm; The update of the augmented reality task offloading scheme based on the particle swarm optimization algorithm involves three factors: intrinsic factors, local factors, and global factors. These correspond to the strategy update method, subtasks, and other aspects, respectively. Uninstallation strategy updates and overall tasks Update the uninstallation strategy; During the wheel unloading strategy update process, subtasks are affected. Corresponding The unloading strategy with the minimum value is the locally optimal strategy, denoted as . ; and thus the overall task Corresponding The unloading strategy with the smallest value is the globally optimal strategy, denoted as . ; Since the update of the unloading strategy is a discrete rather than continuous process, a particle update scheme as shown in formula (11) was designed using the genetic algorithm. (11); in: (12); Indicates when the policy update factor Greater than random number At that time, for the uninstallation scheme Perform single-point mutation operation ,Right now: In the plan In the middle, change any subtask execution platform ; (13); This indicates that the roulette wheel algorithm is used when the subtask policy update factor... Greater than random number Then, make the subtask The corresponding task unloading scheme with the shortest latency and the modified uninstallation scheme Perform single-point crossover operation ,Right now: Indicates the uninstallation scheme In the middle, sub-tasks The execution platform was replaced with The corresponding execution platform ; (14); This indicates that the roulette wheel algorithm is used when the total task update factor is... Greater than random number At that time, in the plan and the globally optimal uninstallation solution In the middle, select any subtask Perform single-point crossover operation ;Right now: Indicates the uninstallation scheme In the middle, sub-tasks The execution platform was replaced with Execution platform for corresponding subtasks ; Will , , Repeat execution in sequence After that, we got the first In the round strategy iteration process Several candidate solutions; subsequently, in this Of the three schemes, the one that maximizes the fitness function shown in formula (10) is selected as the task unloading scheme for the next round of policy iteration. ; Repeat Steps 5.2 and 5.3 until the fitness function is achieved. Convergence to threshold After that, the corresponding Uninstallation solution This is the final augmented reality task offloading solution. After deploying this solution to the corresponding mobile terminal or edge service platform, it can ensure the shortest latency during the task offloading process.

Citation Information

Patent Citations

  • Relay edge network task unloading method based on digital twinning

    CN113590232A

  • Virtual machine service quality dynamic prediction method based on Bayesian network migration

    CN112882917A

  • User computing task unloading method in mobile edge computing

    CN113220356A