Internet of vehicles edge computing multi-target unloading method and system fusing dynamic environment modeling and improved SARSA

Abstract

The invention relates to an Internet of Vehicles edge computing multi-target unloading method and system fusing dynamic environment modeling and improved SARSA, and belongs to the field of intelligent traffic and edge computing fusion. The method and the system comprise MEC environment perception and multi-dimensional state construction, dynamic reward feedback oriented to multi-dimensional performance indexes, intelligent decision model construction and learning based on improved SARSA, and antagonism training oriented to real disturbance. A high-fidelity environment model is constructed through a space-time attention mechanism, an SARSA algorithm is improved to realize hierarchical qualification trace attenuation and collaborative Q table updating, a multi-target hierarchical reward engine is combined to implement differential optimization on an emergency task and a conventional task, and an adversarial training mechanism is introduced to improve robustness. The core problems of high mobility, task diversity, resource limitation and the like in the Internet of Vehicles are effectively solved, the comprehensive performance is optimal in multiple dimensions of delay, energy consumption, resource utilization rate and the like, and the actual landing of the edge computing technology of the Internet of Vehicles is promoted.

Description

Technical Field

[0001] The present invention belongs to the field of integration of intelligent transportation and edge computing, and relates to a multi-target offloading method and system for Internet of Vehicles edge computing that integrates dynamic environment modeling and improved SARSA. Background Art

[0002] Amidst the wave of digitalization and the rapid evolution of intelligent transportation, the convergence of the Internet of Vehicles (IoV) and edge computing has become a core driving force behind the development of next-generation intelligent transportation systems. Through V2V (vehicle-to-vehicle) and V2I (vehicle-to-infrastructure) communication architectures, IoV establishes a dynamic, interactive information network, providing a data exchange foundation for scenarios such as intelligent driving decisions, real-time road condition monitoring, and in-vehicle entertainment services. Edge computing, on the other hand, shifts computing resources to the edge of the network, significantly reducing system latency and improving response speed by shortening data transmission distances. The combination of the two is considered a key path to addressing the high computing demands and low-latency constraints of in-vehicle applications. However, the deep integration of IoV and edge computing presents multiple challenges to the system, stemming from vehicle mobility, diverse task types, and resource constraints. These challenges make traditional resource allocation and task offloading strategies difficult to meet real-world needs.

[0003] The high mobility of vehicles poses a primary dynamic challenge for IoV edge computing systems. On highways, vehicles travel continuously at speeds of 60-120 km / h, resulting in rapidly changing connection states with edge servers. When a vehicle rapidly exits the coverage area of ​​an edge server, the interruption of the communication link directly leads to task processing interruption or the need for re-offloading, posing a serious threat to applications with extremely high real-time requirements, such as emergency braking warnings. For example, a vehicle traveling at 100 km / h travels approximately 1.67 kilometers per minute. If the coverage radius of an edge server is 1 kilometer, the vehicle spends only approximately 36 seconds within the coverage area of ​​a single server. This rapid movement causes frequent changes in the network topology within a short period of time. Traditional static resource allocation strategies, lacking predictive capabilities for vehicle movement and dynamic adjustment mechanisms, struggle to maintain stable task processing links amidst such rapid topological changes. Consequently, task offloading decisions often face the dilemma of server selection lagging behind vehicle location changes, leading to increased data transmission latency or task failure.

[0004] The complexity of resource allocation brought about by task diversity further exacerbates the difficulty of system optimization. In automotive applications, different tasks exhibit significant differences in their computing requirements, data size, and latency constraints. For example, environmental perception tasks in autonomous driving require real-time processing of data from multiple sensors, such as lidar and cameras. This computational effort can reach several gigabytes per second, and processing must be completed within 50ms to ensure driving safety. Meanwhile, map updates in in-car entertainment systems, while potentially consuming hundreds of megabytes of data, tolerate processing delays of several seconds. This heterogeneous task demand forces edge server resources to dynamically balance between compute-intensive tasks (such as obstacle detection), data-intensive tasks (such as HD map downloads), and latency-sensitive tasks (such as collision warning). Traditional heuristic algorithms typically make decisions based on a single metric (such as minimizing latency or maximizing resource utilization), making it difficult to find the optimal balance between these conflicting multi-dimensional objectives. For example, if latency is the sole objective, the system may offload a large number of tasks to the nearest edge server, overloading it and impacting all dependent tasks. Alternatively, excessive focus on resource balancing may cause latency-sensitive tasks to timeout due to extended transmission paths.

[0005] Limited edge resources and uneven load pose bottlenecks to system performance. Edge servers, constrained by hardware specifications, have upper limits on their computing power, storage capacity, and bandwidth. When multiple vehicles simultaneously offload compute-intensive tasks to the same edge node, the server can easily become overloaded, significantly increasing task queuing delays. For example, assuming a single edge server has a computing capacity of 5 units, if three vehicles simultaneously offload 3 units of tasks, the total load reaches 9 units, far exceeding the server capacity. Task processing delays can surge from a normal 10ms to over 50ms, potentially causing tasks to timeout and fail. Furthermore, the random nature of vehicle movement causes temporal and spatial imbalances in the load on edge servers. During peak hours, edge nodes in central city areas may be overloaded, while suburban nodes may be idle. This imbalance further reduces overall system resource utilization. Traditional load balancing strategies, lacking dynamic awareness of vehicle movement patterns and task generation patterns, struggle to implement cross-regional resource scheduling, resulting in a coexistence of local overload and global resource waste.

[0006] In the exploration of technologies to address these challenges, reinforcement learning, as a method capable of autonomously learning optimal policies in dynamic environments, has demonstrated significant advantages. The SARSA algorithm, due to its online learning properties and continuous decision-making capabilities, has been widely used in edge computing task offloading scenarios. However, the traditional SARSA algorithm has exposed several shortcomings in the connected vehicle (IoV) environment. Its ε-greedy strategy produces high-variance update targets during exploration. Especially when ε is high, the fluctuations in random actions can severely impact learning stability. In IoV task offloading scenarios, this randomness can lead to tasks being incorrectly offloaded to distant servers or the cloud, increasing transmission latency. For example, when ε = 0.3, the algorithm has a 30% probability of selecting a non-optimal action. If the optimal action is to offload to a local edge server (transmission latency of 10ms), but the random action is to go to the cloud (transmission latency of 50ms), a single decision could increase latency by 400%. Furthermore, the eligibility trace decay mechanism of the traditional SARSA algorithm uses a uniform parameter for all action types, failing to distinguish the long-term impact of different actions, such as local processing, edge offloading, and cloud offloading. This results in the algorithm's limited adaptability to both urgent and routine tasks.

[0007] More critically, the reward function design of the traditional SARSA algorithm usually focuses on a single goal (such as minimizing latency), which makes it difficult to balance the multi-dimensional optimization needs in the Internet of Vehicles scenario. In practical applications, goals such as latency, energy consumption, resource utilization, and load balancing are often mutually constrained - reducing latency may require offloading tasks to edge servers with greater computing power but higher energy consumption, while optimizing resource utilization may require sacrificing a certain amount of latency. This single-goal-oriented reward function will cause the algorithm's performance on non-optimization goals to degrade significantly. For example, an algorithm that only uses latency as an optimization goal may frequently offload tasks to a close but highly loaded edge server, causing the server to overload, which in turn causes chain delays, which goes against the original intention of the optimization.

[0008] To address the shortcomings of traditional algorithms, researchers have addressed improvements in environmental modeling, algorithm optimization, and reward design. In environmental modeling, the introduction of a spatiotemporal attention mechanism is key to improving dynamic perception capabilities. Spatial attention dynamically adjusts server service priorities using Euclidean distance weighting, giving higher weight to servers closer to the vehicle. Temporal attention applies decaying weights to historical load sequences, enhancing the system's responsiveness to sudden load fluctuations. This modeling approach effectively addresses the dynamic topological changes caused by high vehicle mobility, enabling the agent to more accurately perceive the environment. At the algorithmic optimization level, techniques such as expected value updating, hierarchical eligibility traces, and collaborative Q-table updating have been proposed. These techniques improve the algorithm's convergence speed and decision stability by reducing update variance, differentiating action-type decay coefficients, and promoting distributed policy consistency. In reward function design, a multi-objective hierarchical reward engine achieves differentiated optimization for urgent and routine tasks. Combined with a load variance penalty, it effectively balances multiple objectives such as latency, energy consumption, and resource balancing.

[0009] However, existing improved methods still have shortcomings when dealing with adversarial scenarios in the Internet of Vehicles. When the edge server is suddenly overloaded or the task deadline is compressed, the system is prone to policy collapse. For example, when the server load is suddenly saturated, traditional algorithms may continue to offload tasks to overloaded nodes due to lack of training for abnormal scenarios, resulting in a large number of tasks timeouts. To address this problem, an adversarial training mechanism was introduced. By simulating scenarios such as server overload and task deadline compression, and using a priority experience replay strategy to strengthen learning in abnormal scenarios, the robustness of the system was significantly improved.

[0010] In summary, IoV edge computing faces multiple challenges during its integration, including high mobility, diverse tasks, and limited resources. Traditional algorithms face shortcomings in dynamic adaptability, multi-objective optimization, and robustness that urgently need to be addressed. Improved approaches based on reinforcement learning, particularly those combining spatiotemporal attention modeling, hierarchical reward design, and adversarial training, offer an effective path to overcoming these bottlenecks. However, continued optimization in areas such as algorithm efficiency and scenario generalization is still needed to promote the practical implementation of IoV edge computing technology. Summary of the Invention

[0011] In view of this, the purpose of the present invention is to provide a multi-target offloading method and system for edge computing in the Internet of Vehicles that integrates dynamic environment modeling and improved SARSA. The method constructs a high-fidelity environment model based on the spatiotemporal attention mechanism, realizes hierarchical qualification trace decay and collaborative Q-table update through the improved SARSA algorithm, and combines a multi-target hierarchical reward engine to implement differentiated optimization of emergency tasks (such as collaborative emergency braking) and routine tasks (such as in-vehicle entertainment). The adversarial training mechanism is introduced to improve the robustness of the system in scenarios with sudden load fluctuations and compressed task deadlines, effectively solving core problems such as network topology changes caused by high mobility in the Internet of Vehicles, conflicts in heterogeneous task resource requirements, and limited computing power of edge nodes.

[0012] In order to achieve the above object, the present invention provides the following technical solutions:

[0013] A multi-target offloading method for edge computing in Internet of Vehicles (IoV) that integrates dynamic environment modeling and improved SARSA, the method comprising:

[0014] Obtain dynamic vehicle network environment parameters and perform spatiotemporal perception multi-dimensional state encoding based on real-time data;

[0015] Establish an adaptive strategy decision model based on improved SARSA, and perform strategy generation and optimization;

[0016] The offloading action is actually executed according to the generated execution strategy, and a multi-dimensional constraint compound reward evaluation is performed based on the actual execution results;

[0017] The model is optimized offline and continuously learned based on real-world disturbance perception, key experience, hierarchical credit allocation, and collaborative Q-table iterative optimization.

[0018] Furthermore, the Internet of Vehicles environment includes at least vehicles, roadside units, edge servers, and cloud servers. The Internet of Vehicles environment parameters include the number of edge servers (max_servers), the upper limit of the computing processing capacity of each edge server node (edge_capacity), the computing processing capacity of the central cloud server (cloud_capacity), the maximum task processing delay threshold (max_delay) required by the quality of service, and the geographical deployment location (server_positions) of each edge server; the length of the historical data analysis window (history_window) and the maximum number of interaction steps in a single decision or evaluation cycle (episode_length) are set;

[0019] At the beginning of each decision cycle, the following information is dynamically acquired or received: the current real-time location of the vehicle user device (vehicle_pos), the amount of data for the newly arrived or currently processed computing task (task_size), and the required processing deadline (task_deadline). Based on task_deadline and the current time, the task is determined to be urgent, and the task is classified as either urgent or ordinary. Urgent tasks are marked as is_urgent = True, and ordinary tasks are marked as is_urgent = False.

[0020] Monitor and obtain the current computing load status of each edge server and cloud server in real time.

[0021] Furthermore, in the process of dynamic change perception of edge offloading tasks in the Internet of Vehicles environment, the following processes are included: spatial correlation analysis, time series trend analysis, spatiotemporal feature fusion and aggregation, and system load balancing measurement, among which,

[0022] During the spatial correlation analysis, the physical distance between the user device and each server is calculated based on the real-time location of the user device and the fixed location of each edge server. The load information of each server is weighted using an exponential decay function to ensure that the closer the server is to the user device, the greater its load influence in the state representation. The spatial weight calculation formula is:

[0023]

[0024] Where, d i is the Euclidean distance vector between the user device and each edge server, i is the index of the vehicle (user device) that currently needs a decision; j is the index of the edge server, and N is the total number of available edge servers

[0025] During the time series trend analysis process, a sliding window load_history containing the load data of each server within the past history_window time steps is maintained. A linearly increasing weight vector is used to perform weighted averaging on the historical edge server load data within the window to extract the historical trend of load changes. The time weight calculation formula is as follows (the sliding window is represented by T in the formula):

[0026]

[0027] In the process of spatiotemporal feature fusion and aggregation, the spatial correlation weight and time series trend are combined to calculate the current edge server's spatially weighted average load current_feature and the historical edge server load data's temporally and spatially weighted average trend feature history_features. The calculation formula is:

[0028]

[0029] Where, F e is the current load vector of each edge server;

[0030] During the system load balancing measurement process, the statistical variance load_variance of the current load of all edge server nodes and central cloud servers is calculated as an indicator to measure whether the load of the entire system is balanced;

[0031] Finally, the task attributes task_size, task_deadline, is_urgent and the environmental features current_feature, history_features, cloud_load, load_variance are combined, and the necessary normalization or standardization is performed on each component to form a fixed-dimensional state vector S t .

[0032] Furthermore, in the adaptive strategy decision model based on the improved SARSA, the continuous or high cardinality state feature S t Discretization is performed according to the binning rule, where predefined interval boundaries are mapped to discrete state indices. Then, a Q-value table q_table and a lattice trace table e_table are initialized. The Q-value table q_table is used to store the estimated value of each discrete state-action pair, and the eligibility trace table e_table is used as an auxiliary structure for subsequent learning optimization.

[0033] In the decision-making process, the agent randomly selects a feasible unloading action with an exploration rate of ε; and selects the action with the maximum Q value in the current discrete state with probability 1-ε;

[0034] Finally, according to the adaptive strategy π(A t |S t ) is the current state S t Select the optimal or suboptimal unloading action A t , where the updates of the Q-value table q_table and the eligibility trace table e_table, as well as the updates of the optimization mechanism depend on the feedback obtained from the environment after the action is executed.

[0035] Further, the final confirmed uninstall action A t The system converts the task into actual instructions, which are executed on the real computing and network infrastructure through the vehicle communication module and network interface. After the actual execution is completed, the system will collect and analyze the various real performance indicators of the task execution, and evaluate the effect of the decision based on the preset multi-objective reward engine to generate a scalar reward signal R t,in,

[0036] For urgent tasks, if the actual completion delay D does not exceed its task deadline D td , then a positive reward is given according to the remaining time margin, and the calculation formula is as follows:

[0037]

[0038] If the timeout is exceeded, a significant negative penalty will be imposed based on the duration of the excess, calculated as follows:

[0039] P ou =C2·max(DD td ,0)

[0040] Where C1 and C2 are the preset reward and penalty coefficients. If the task is processed locally on the vehicle, the actual energy consumption energy_actual is deducted from the total reward according to a certain coefficient C3.

[0041] For non-urgent tasks, we encourage offloading them to servers with lower actual loads. The reward is positively correlated with the estimated or actual remaining available resource ratio of the target server after offloading. The calculation formula is:

[0042] R rn =C4·(1-U)

[0043] Completion on time is also rewarded, but its weight is lower than that of urgent tasks. The formula is:

[0044]

[0045] Where C5 is the preset non-urgent task timeliness reward coefficient; the actual energy consumption of local processing, energy_actual, is penalized with an increased penalty weight and deducted from the total reward by coefficient C6;

[0046] A global load variance penalty term is introduced to promote load balancing. This penalty is calculated based on the load variance load_variance (σ is used here) calculated based on the real-time load of all servers. 2 is proportional to:

[0047] R balance =C7·σ 2

[0048] Where C7 is the load variance penalty coefficient;

[0049] Considering that extreme events that may occur in the real environment may cause the reward signal to fluctuate violently, the calculated single-step reward r t impose a lower limit;

[0050] The above reward / penalty components are weighted and summed by the preset weight coefficients according to the importance of each optimization goal in the actual application scenario to obtain the final composite reward signal R t .

[0051] Furthermore, in the optimization process of the adaptive strategy decision model based on the improved SARSA, the eligibility trace mechanism is introduced to accelerate the learning process and effectively allocate credits. The attenuation factor λ of the eligibility trace is dynamically adjusted according to the type of the currently selected action. The update rule of the eligibility trace is as follows:

[0052]

[0053] The experience tuple (S t ,A t ,R t+1 ,S t+1 ,done) is stored in an experience replay buffer; if the experience is generated under a simulated adversarial event, the experience is given a higher storage priority;

[0054] Periodically extract a batch of experience data from the experience playback buffer, for each experience (s t ,a j ,r j ,s j ′,d j ),have:

[0055] Calculate the next state s j ′, the expected Q value that can be obtained by following the current ε-greedy strategy:

[0056]

[0057] Sum all possible actions a, where π(a|s {j+1,d} ) is in state s {j+1,d} The probability of choosing action a;

[0058] Calculate the time difference error:

[0059]

[0060] Where γ is the discount factor, s{ j,d} is state s j Discretization representation of ;

[0061] Update the qualification trace table E(s d ,a): For all state-action pairs (s d ,a)Execute:

[0062] E(s d,a)=γ·λ·E(s d ,a)

[0063] The state-action pair corresponding to the current experience (s {j,d} ,a j )implement:

[0064] E(s {j,d} ,a j )=E(s {j,d} ,a j )+1

[0065] S354, update Q value table Q(s d ,a): For all state-action pairs (s d ,a), execute:

[0066] Q(s d ,a)=Q(s d ,a)+α·δ·E(s d ,a)

[0067] Where α is the learning rate.

[0068] Finally, every fixed update period, a smoothing operation is performed on the Q-values ​​of the actions related to offloading to different edge servers in the Q-value table to converge to the mean.

[0069] Furthermore, an adversarial event simulation mechanism is introduced during the training phase, and simulated adversarial scenarios are used to enhance the offline or semi-online training phase. The adversarial event types are defined as:

[0070] Sudden high load / downtime of edge servers: In the simulation environment, one or more edge servers are randomly selected with a certain probability adv_prob, and their available computing resources edge_capacity are instantly reduced to a very low level or completely unavailable;

[0071] Sudden change in task urgency: The deadline of the simulated task, task_deadline, is suddenly brought forward;

[0072] Then, based on the recorded real disturbance pattern or the preset typical failure scenario, a simulated adversarial stress test event is injected with a certain probability adv_prob;

[0073] After perceiving real sudden interference or injecting simulated adversarial events into training, the resulting states, actions, rewards, next states, and completion flags will be stored and utilized using special strategies. If the generation of an experience is associated with the type of adversarial event, this key / experience will be given a higher learning priority in the experience replay pool.

[0074] In the process of sampling experience from the experience replay pool and updating the Q-table, an advanced Q-table iteration mechanism is applied to further optimize credit allocation and accelerate knowledge sharing:

[0075] First, through a hierarchical credit allocation mechanism, which dynamically selects different eligibility trace decay coefficients λ based on the action type, we ensure that the obtained real reward signal can be traced back to the historical decisions that contributed to the result;

[0076] Secondly, through the collaborative Q-table iteration mechanism, every preset update interval, the Q-values ​​of all edge server offloading-related actions in the Q-value table are subjected to mean regression based on the current Q-table state: the specific operations are:

[0077]

[0078] Among them, w is the regression weight, is the mean Q value of all edge offloading actions in the current state;

[0079] Through the combined effect of the above mechanisms, the speed at which Q-value learning can adapt to the dynamics of the real environment and the performance of the final strategy in practical applications can be improved.

[0080] On the other hand, a system for executing the aforementioned multi-target offloading method of edge computing of Internet of Vehicles integrating dynamic environment modeling and improved SARSA is proposed, characterized in that: the system includes a MEC environment perception and multi-dimensional state construction module, a dynamic reward feedback mechanism module for multi-dimensional performance indicators, an intelligent decision model construction and learning module based on improved SARSA, and an adversarial training module for real-world disturbances, wherein,

[0081] The MEC environment perception and multi-dimensional state construction module is used to fully perceive the edge computing environment of the Internet of Vehicles and construct a multi-dimensional state vector for reinforcement learning decision-making. It collects vehicle location, task attributes, and load information of edge servers and cloud servers in real time, and combines spatial and temporal attention mechanisms to encode the environmental state. It generates a multi-dimensional state tuple that fully reflects the current environmental conditions, providing a basis for subsequent intelligent decision-making.

[0082] The dynamic reward feedback mechanism module for multi-dimensional performance indicators designs a dynamic reward function that can reflect multi-dimensional performance goals. It provides positive or negative numerical feedback to the agent based on the actual results of each task offloading decision. It dynamically calculates and generates reward signals based on task processing delay, system energy consumption, resource utilization, and load balancing to guide the agent to learn offloading strategies that meet actual application needs.

[0083] The intelligent decision-making model construction and learning module based on the improved SARSA is used to build and optimize the task offloading decision model based on the enhanced state-action-reward-state-action reinforcement learning algorithm. By introducing hierarchical eligibility trace decay, collaborative Q-table updates, and expected SARSA update rules, it reduces the Q-value update variance, improves the algorithm convergence speed and long-term reward propagation efficiency, and enhances distributed policy consistency.

[0084] The adversarial training module for real-world disturbances is used to improve the robustness of intelligent agents in responding to emergencies in actual deployments. By simulating adversarial event scenarios and combining experience replay priority strategies with policy entropy regularization, the intelligent agent is trained to enhance its ability to learn and adapt to abnormal scenarios and reduce the risks of policy collapse and overfitting.

[0085] A computer-readable storage medium is also proposed, characterized in that a computer program is stored on the storage medium. When the computer program is executed by a processor, the aforementioned multi-target offloading method of Internet of Vehicles edge computing that integrates dynamic environment modeling and improved SARSA is implemented.

[0086] A computer program product is also proposed, comprising a computer program, which, when executed by a processor, implements the aforementioned multi-target offloading method for edge computing of the Internet of Vehicles that integrates dynamic environment modeling and improved SARSA.

[0087] The beneficial effects of the present invention are:

[0088] This paper addresses the core challenges of high mobility, multi-tasking concurrency, and resource heterogeneity in edge computing scenarios in the Internet of Vehicles (IoV). It proposes a multi-objective dynamic resource offloading method and system based on improved reinforcement learning. Through an innovative algorithm architecture and collaborative optimization mechanism, it significantly improves resource scheduling efficiency and system robustness. The specific advantages are as follows:

[0089] First: This paper proposes a dynamic environment modeling method that integrates spatiotemporal attention. It dynamically allocates the service weights of vehicles and edge servers through the spatial attention module, and combines the temporal attention module to perform attenuation weighting on historical load sequences to construct a high-fidelity state representation model. This effectively shortens the sudden load response delay and significantly improves the environment modeling accuracy and local overload prediction capabilities.

[0090] Second: This paper designs a hierarchical reinforcement learning optimization framework and adopts an improved SARSA algorithm to replace the traditional mechanism. Through strategy weighted updates and hierarchical eligibility trace design, it significantly reduces the Q-value update variance, improves the algorithm convergence speed and long-term reward propagation efficiency, and enhances the consistency of distributed strategies.

[0091] Third: The present invention constructs a multi-objective hierarchical reward engine, introduces a dynamic penalty mechanism for emergency tasks, combines resource utilization rewards with load variance penalties, and realizes the coordinated optimization of latency, energy consumption and load balancing, significantly improving the timeliness of emergency task processing and optimizing resource utilization and system-level load balancing performance.

[0092] Fourth: This invention proposes a dynamic defense scheme for adversarial events. By simulating server overload attacks and task deadline compression to build an adversarial training environment, combined with the experience replay priority strategy and policy entropy regularization, it effectively reduces the risk of policy collapse and overfitting, and significantly improves system reliability and task transmission success rate.

[0093] Fifth: The present invention realizes intelligent cross-domain resource collaborative scheduling, reserves edge resources in advance based on trajectory prediction, optimizes state space modeling through dynamic state discretization technology, significantly compresses server switching delay fluctuations, and improves cross-domain task migration efficiency and spectrum utilization capabilities.

[0094] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0096] Figure 1 This is a schematic diagram of the overall process of a multi-target offloading method for edge computing in an Internet of Vehicles (IoV) that integrates dynamic environment modeling and improved SARSA under an embodiment of the present invention;

[0097] Figure 2 This is a schematic diagram of a cyclic training process of a multi-target offloading method for edge computing in an Internet of Vehicles (IoV) that integrates dynamic environment modeling and improved SARSA under an embodiment of the present invention. DETAILED DESCRIPTION

[0098] The following describes the embodiments of the present invention by means of specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0099] Among them, the accompanying drawings are only for illustrative purposes and represent only schematic diagrams rather than actual pictures, and should not be understood as limiting the present invention. In order to better illustrate the embodiments of the present invention, some parts of the accompanying drawings may be omitted, enlarged or reduced, and do not represent the dimensions of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions may be omitted in the accompanying drawings.

[0100] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "left", "right", "front", "back", etc. indicating directions or positional relationships, they are based on the directions or positional relationships shown in the drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operate in a specific direction. Therefore, the terms describing the positional relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0101] See also Figures 1 and 2 , which is a multi-target offloading method and system for Internet of Vehicles edge computing that integrates dynamic environment modeling and improved SARSA.

[0102] Example 1

[0103] This embodiment aims to optimize dynamic task offloading and resource management in a mobile edge computing (MEC) environment. It proposes an intelligent decision-making framework based on reinforcement learning that integrates multidimensional environmental perception, adversarial training, and an enhanced SARSA algorithm. By constructing an EnhancedMECEnv environment that includes a spatiotemporal attention mechanism, the model can capture the complex state characteristics of the MEC system, such as the spatial distribution and temporal evolution trend of server load. To improve the robustness of the model under real-world perturbations, an adversarial event injection mechanism is introduced into the environment to simulate situations such as sudden high server load or sudden changes in the urgency of task requirements. The intelligent agent EnhancedSarsaAgent adopts a prioritized experience replay strategy to prioritize learning experience in adversarial scenarios. It also combines hierarchical eligibility traces (dynamically adjusting the λ value based on the action type) and expected SARSA update rules to accelerate learning and improve the long-term effectiveness of decisions. In addition, to promote collaboration and load balancing among edge servers, the intelligent agent also integrates a collaborative Q-value update mechanism. The technical solution of the present invention aims to train an intelligent management model through the collaborative work of the above components, which can autonomously make efficient task offloading decisions in a dynamic and uncertain MEC environment and achieve optimal comprehensive performance in multiple dimensions such as latency, energy consumption, and resource utilization.

[0104] The overall process is as follows Figure 1 As shown, it specifically includes the following steps:

[0105] Step 1: Initialize dynamic vehicle networking environment parameters

[0106] Step 1.1: After the system is deployed and started, it first senses and acquires key dynamic parameters of the current connected vehicle environment in real time through onboard sensors (such as BeiDou and vehicle status sensors), communication with roadside units (RSUs), and interfaces with edge / cloud management platforms. Static deployment parameters include: the number of deployed edge servers, denoted as max_servers; the computing capacity of the cloud server, denoted as cloud_capacity; the computing capacity of each edge server, denoted as edge_capacity; and the precise geographic location of each edge server, denoted as server_positions. Furthermore, the maximum task processing delay tolerance required by the system design or application is defined, denoted as max_delay, and the number of cycle steps in the internal decision logic, denoted as episode_length.

[0107] Step 1.2. At the beginning of each decision cycle (or when a task arrives), the system will dynamically obtain or receive the following information: the current real-time position of the vehicle, recorded as vehicle_pos; the amount of data of the newly arrived or currently processed computing task, recorded as task_size (provided by the on-board application or upper-layer service); the required processing deadline of the task, recorded as task_deadline (specified by the application layer); and based on task_deadline and the current time, determine whether the task is an urgent task, recorded as is_urgent.

[0108] Step 1.3: Simultaneously, the system monitors and obtains the current computing load of each edge server and cloud server in real time (e.g., by periodically querying or subscribing to each node's monitoring API). These real-time parameters together form the dynamic foundation for all subsequent decision-making and learning.

[0109] Step 2: Spatiotemporal perception multi-dimensional state coding based on real-time data

[0110] Step 2.1: To enable the agent to make accurate decisions based on the latest environmental information, this step employs multidimensional state encoding technology and incorporates deep perception of the environment's spatiotemporal dynamics. First, a spatial attention mechanism is introduced: Based on the Euclidean distance between the vehicle's current position (vehicle_pos) obtained in real time in Step 1.2 and the edge server positions (server_positions) known in Step 1.1, a dynamic, exponentially decaying weight vector is calculated. This weighting gives higher weight to the real-time load information of edge servers geographically closer to the vehicle in the state representation.

[0111] Step 2.2. Next, introduce a temporal attention mechanism: the system maintains a sliding window of each server's historical load, denoted as load_history (this historical data comes from the continuous monitoring in step 1.3). A linearly increasing weight vector is used to weight the historical server load data within the window. This mechanism ensures that recently monitored load change trends contribute more to the representation of the current state.

[0112] Step 2.3. Finally, integrate the task_size, task_deadline, and is_urgent attributes of the current task obtained in step 1.2; the current average load of each edge server after spatial attention weighting, recorded as current_feature; the trend characteristics of the historical server load after time and space dual attention weighting, recorded as history_features; the currently monitored cloud server load, recorded as cloud_load; and the load variance calculated based on the current real-time load of all servers, recorded as load_variance. Together, these information constitute a comprehensive, multi-dimensional enhanced state tuple S that reflects the current real environment status. t .

[0113] Step 3: Generation and optimization of adaptive strategies based on improved SARSA

[0114] Step 3.1. The core of this step is to establish a basic framework for adaptive decision-making and select an action based on this framework for the current environment state. This decision-making framework is based on the improved SARSA algorithm. First, the system discretizes the state features obtained from step 2.3 (especially the continuous or multi-valued parts) using a preset binning rule to construct a discrete state space. On this basis, a Q-value table, denoted as q_table, is initialized to store the estimated value of each discrete state-action pair, as well as an eligibility trace table, denoted as e_table, which serves as an auxiliary structure for subsequent learning optimization.

[0115] Step 3.2: Action selection adopts ε-greedy strategy. The agent selects the action based on the state tuple S obtained from step 2.3. t , randomly selects an available action for exploration with probability epsilon, and selects the action with the maximum Q value in the current discrete state with probability 1-epsilon for use. This selected action is . The exploration rate epsilon will adaptively decay as the learning progresses and is guaranteed to be no less than a minimum exploration rate min_epsilon. (Note: The specific numerical updates of the Q value table q_table and the eligibility trace table e_table, as well as more advanced optimization mechanisms such as hierarchical lambda values ​​and collaborative updates, all rely on the feedback obtained from the environment after the action is executed, that is, the reward R t+1 and the next state S t+1 Therefore, the detailed process of these learning and updating will be described in step 5 and subsequent steps.)

[0116] Step 4: Actual Implementation and Scheduling of Heterogeneous Resource Load Balancing

[0117] Step 4.1: Adaptive strategy π(A) generated in step 3 t |St ), the agent is in the current state S obtained from step 2.3 t Select the optimal or suboptimal unloading action A t .

[0118] In step 4.2, the action (local processing, offloading to a specific edge server, or offloading to a cloud server) is then translated by the system into actual instructions, which are executed on the actual computing and network infrastructure through the vehicle's communication module and network interface. For example, if the decision is to offload to the edge, the task data will be transmitted to the target edge node and its computing resources will be requested. During execution, the computing load of the relevant servers will change due to the actual processing of the task, and these changes will be subsequently detected through the monitoring mechanism in step 1.3.

[0119] Step 5: Multi-dimensional constraint compound reward evaluation based on actual execution results.

[0120] Step 5.1: After the task is actually executed in step 4, the system will collect and analyze the actual performance indicators of the task execution, and evaluate the effect of the decision based on the preset multi-objective reward engine to generate a scalar reward signal R t .

[0121] Step 5.2: For tasks marked as urgent, establish a strong reward and penalty mechanism: rewards for on-time completion are positively correlated with the actual measured proportion of remaining processing time, calculated as follows:

[0122]

[0123] If the timeout is exceeded, a severe penalty proportional to the actual timeout duration will be imposed. The calculation formula is:

[0124] P ou =C2·max(DD td ,0)

[0125] At the same time, if the task is processed locally in the vehicle, the actual energy consumption energy_actual (which can be estimated or measured by the on-board power management unit) will be deducted from the total reward according to a certain coefficient C3.

[0126] Step 5.3: For non-urgent tasks, the reward mechanism focuses on resource utilization efficiency and energy conservation: offloading tasks to servers with lower actual loads is encouraged. The reward is positively correlated with the estimated or actual remaining available resources of the target server after offloading. The calculation formula is:

[0127] R rn =C4·(1-U)

[0128] Completion on time is also rewarded, but its weight is lower than that of urgent tasks. The formula is:

[0129]

[0130] Among them, C5 is the preset non-urgent task timeliness reward coefficient; the actual energy consumption energy_actual penalty weight of local processing increases and will be deducted from the total reward according to the coefficient C6.

[0131] Step 5.4: To promote load balancing of the entire MEC system, a global load variance penalty is introduced. This penalty is proportional to the load variance load_variance calculated based on the current real-time load of all servers (derived from the status information in step 2.3 or recalculated):

[0132] R balance =C7·σ 2

[0133] Step 5.5: Considering that extreme events may occur in the real environment and cause the reward signal to fluctuate violently, this module sets a lower limit for the total reward value to maintain the stability of the learning process.

[0134] Step 5.6: At this point, R is calculated from step 5. t+1 , experience tuple (S t , A t , R t+1 , S t+1 , done) has been formed. For the expected SARSA algorithm, the agent will also be based on the new state and the current strategy S t+1 And the current policy π predicts the expected Q value of all possible next actions. TD error δ t Calculated as:

[0135]

[0136] To optimize learning efficiency, a hierarchical lambda value mechanism is introduced. Depending on the action type selected by the agent (local processing, cloud processing, edge processing), different eligibility trace decay coefficients λ are dynamically selected. Therefore, the update rules of the eligibility trace are as follows:

[0137]

[0138] The Q table is updated as follows:

[0139] Q(s,a)←Q(s,a)+αδE t (s,a)

[0140] Step 6: Real-world disturbance perception and robustness training scenario construction

[0141] Step 6.1: To improve the decision-making robustness of the intelligent agent in dealing with non-stationary and uncertain environments in actual deployment, this step emphasizes learning from sudden interference or unexpected events perceived in the real environment, and can be supplemented by simulated adversarial scenarios during the offline or semi-online training phase.

[0142] Step 6.2: During actual operation, the system continuously monitors key performance indicators and resource status. If it detects "real adversarial" events, such as abnormal edge server response (e.g., load instantaneously reaching its upper capacity edge_capacity, information derived from the monitoring in Step 1.3), a sharp drop in network link quality, or a sudden change in task requirements (e.g., task_deadline) due to external factors, these events and the resulting decision consequences will be specifically flagged.

[0143] Step 6.3: During the continuous learning or offline optimization phase of the model, simulated adversarial stress test events can be injected with a certain probability adv_prob (this probability can be dynamically adjusted according to the training strategy) based on the recorded real disturbance patterns or preset typical failure scenarios to encourage the agent to learn a more comprehensive response strategy.

[0144] Step 7: Prioritize reinforcement learning of key experiences (especially adversarial ones)

[0145] Step 7.1: After a real sudden disturbance is perceived in step 6 or a simulated adversarial event is injected during training, the relevant experience (state, action, reward, next state, completion flag) generated will adopt a special storage and utilization strategy.

[0146] Step 7.2: If an experience is associated with a real adversarial event marked in step 6.2 or a simulated adversarial event injected in step 6.3, then this "critical / adversarial experience" is given a higher learning priority in the experience replay pool. This can be achieved by, for example, storing it multiple times (the number of repetitions is controlled by the parameter adversarial_proiority) or giving it a higher selection probability during sampling.

[0147] Step 7.3: Ordinary, routine experiences are stored and sampled in a standard manner. By reinforcing learning from critical negative or unexpected scenarios, the agent can more quickly master the ability to cope with the complexities of the real world.

[0148] Step 8: Layered credit allocation and collaborative Q-table iterative optimization

[0149] Step 8.1: In the process of sampling experience from the experience replay pool and updating the Q-table, this step applies an advanced Q-table iteration mechanism to further optimize credit allocation and accelerate knowledge sharing to adapt to the complexity of the real environment.

[0150] Step 8.2: First, through a hierarchical credit assignment mechanism, that is, dynamically selecting different eligibility trace decay coefficients λ according to the action type (local, edge, or cloud), the true reward signal obtained from step 5 can be more accurately and effectively traced back to the historical decisions that contributed to the result.

[0151] Step 8.3. Secondly, through the collaborative Q-table iteration mechanism, every preset update interval (this interval can be adjusted to follow the system learning stability and environmental change rate), the Q-values ​​of all edge server offload-related actions in the Q-value table are subjected to mean regression based on the current Q-table state. The specific operations can be:

[0152]

[0153] Where w is the regression weight, is the mean Q-value of all edge offloading actions in the current state. This promotes the knowledge transfer and generalization ability of the agent between similar (but possibly heterogeneous) real edge nodes.

[0154] These mechanisms work together to improve the speed at which Q-value learning adapts to the dynamics of the real environment and the performance of the final strategy in practical applications.

[0155] Example 2

[0156] This embodiment describes a dynamic task offloading and resource intelligent management system based on reinforcement learning applied to mobile edge computing (MEC) scenarios.

[0157] The core process of this embodiment is as follows Figure 2 As shown: The system includes:

[0158] MEC environment perception and multi-dimensional state building modules:

[0159] In this embodiment, the MEC environment perception and multi-dimensional state construction module executes step S1. Step S1 is intended to fully perceive the MEC environment and construct a multi-dimensional state vector for reinforcement learning decision-making. Specifically, it includes the following sub-steps:

[0160] S11: First, parameterize the target MEC system and configure key operating parameters, including but not limited to: the number of available edge server nodes (max_servers), the upper limit of the computing processing capacity of each edge server node (edge_capacity), the computing processing capacity of the central cloud server (cloud_capacity), and the maximum task processing delay threshold (max_delay) required by the Quality of Service (QoS). At the same time, record the geographical deployment location (server_positions) of each edge server. To achieve dynamic data collection, establish an interface with the MEC system monitoring platform to obtain real-time operating data. Set the length of the historical data analysis window (history_window) and the maximum number of interaction steps in a single decision or evaluation cycle (episode_length).

[0161] S12: At each decision time point or when a new computing task request arrives, the following data is collected in real time through the MEC system monitoring interface:

[0162] The current precise geographic location of the user's device, vehicle_pos.

[0163] Key attributes of newly arrived computing tasks include the amount of data the task needs to process (task_size, e.g., MB) and the task's required final completion deadline (task_deadline, e.g., milliseconds). Based on the urgency of the deadline, tasks are classified as urgent (marked with is_urgent = True) or normal (marked with is_urgent = False).

[0164] The current real-time computing resource occupancy rate or remaining available capacity of each edge server node and central cloud server is used to obtain the current computing load of each server.

[0165] S13: To fully capture the dynamic changes in the MEC environment, the following techniques are used for feature engineering:

[0166] Spatial Correlation Analysis Module: Based on the real-time location of the user device and the fixed location of each edge server, the physical distance between the user device and each server is calculated. An exponential decay function is used to assign weights to the load information of each server, so that the closer the server is to the user device, the greater the impact of its load status in the state representation. The specific spatial weight calculation formula is:

[0167]

[0168] where d i is the Euclidean distance vector between the user device and each server.

[0169] Time Series Trend Analysis Module: This module maintains a sliding window (load_history) containing the load data of each server within the past history_window time steps. A weighted average method is used on the data within the window to extract historical trends in load changes.

[0170] Spatiotemporal feature fusion and aggregation: Combining the spatial correlation weight and time series trend, we calculate the current spatially weighted average load of each edge server, current_feature, which is calculated as follows:

[0171]

[0172] Among them F e is the current load vector of each edge server; and history_features is the average trend feature of the historical server load data after time and space dual weighting.

[0173] System load balancing measurement module: Calculates the statistical variance load_variance of the current load of all edge server nodes and central cloud servers as an indicator to measure whether the load of the entire system is balanced.

[0174] S14: Combine the task attributes (task_size, task_deadline, is_urgent) extracted in sub-step (2) with the environmental features (current_feature, history_features, cloud_load, load_variance) calculated in sub-step (3), and perform necessary normalization or standardization on each component to form a fixed-dimensional state vector S t The vector S t It will serve as the main basis for subsequent reinforcement learning agent decision-making.

[0175] Dynamic reward feedback mechanism for multi-dimensional performance indicators:

[0176] In this embodiment, step S2 aims to design a dynamic reward function that can reflect multi-dimensional performance objectives. Based on the actual effect of each task offloading decision, positive or negative numerical feedback is given to the agent to guide the agent to learn an offloading strategy that meets actual application requirements. Specifically, it includes the following sub-steps:

[0177] S21: Task processing delay (QoS) evaluation:

[0178] For urgent tasks, if the actual completion delay D does not exceed its task deadline D td , then a positive reward is given according to the remaining time margin, and the calculation formula is as follows:

[0179]

[0180] If the timeout is exceeded, a significant negative penalty will be imposed based on the duration of the excess, calculated as follows:

[0181] P ou =C l2U ·max(DD td ,0)

[0182] Among them C l1U and C l2U are the preset reward and penalty coefficients.

[0183] For ordinary tasks, the weight of delay-related rewards and penalties is relatively low.

[0184] S22: System energy consumption assessment:

[0185] If the task is processed locally on the user device, a negative reward (penalty) is applied based on the estimated energy consumption E, calculated as follows:

[0186] P e =C e1 ·E

[0187] Among them C e1 is the energy consumption penalty coefficient.

[0188] If the task is offloaded to the server, the data transmission energy consumption and server processing energy consumption are considered. For non-urgent tasks, the weight of energy penalty C e2 The corresponding weight can be appropriately higher than that of urgent tasks.

[0189] S23: Resource utilization and load balancing evaluation:

[0190] To encourage offloading tasks to servers with lower current loads, the reward value can be proportional to the percentage of remaining available resources on the target server, calculated as follows:

[0191] R rn =C r1 (1-U)

[0192] Among them C r1 The resource utilization reward coefficient.

[0193] A negative reward (penalty) is imposed on the load variance of the entire system to suppress excessive load concentration. The calculation formula is as follows:

[0194]

[0195] Among them C lb1 is the load balancing penalty coefficient.

[0196] S24: After the simulated adversarial event (such as sudden server failure or network congestion, see step S4 for details) occurs, in order to avoid the violent oscillation of the reward signal from damaging the learning process, the calculated single-step reward r t A reasonable lower bound is imposed to enhance the robustness of the model.

[0197] S25: The reward / penalty components of the above performance dimensions are weighted and summed by the preset weight coefficients according to the importance of each optimization goal in the actual application scenario to obtain the final composite reward signal R t These weight coefficients can be adjusted according to actual results during the system debugging phase.

[0198] Intelligent decision-making model construction and online learning based on improved SARSA:

[0199] In this embodiment, step S3 uses an enhanced state-action-reward-state-action (SARSA) reinforcement learning algorithm (hereinafter referred to as EnhancedSarsaAgent) to build and optimize the task offloading decision model, which specifically includes the following sub-steps:

[0200] S31: To apply tabular Q-learning, the continuous or high-cardinality state features constructed in step S1 are mapped to discrete state indices based on predefined bin boundaries (stored in agent.bins ). This step ensures that the dimensions of the Q-value table (q_table) and the eligibility trace table (e_table) are manageable.

[0201] S32: During the decision-making process, the agent randomly selects a feasible unloading action with probability ε (exploration rate) to explore unknown effects of the action. It also selects the action with the maximum Q value in the current discrete state with probability 1-ε to utilize the learned optimal policy. The exploration rate ε is set to a large value at the beginning of training and gradually decays to a small stable value as training progresses.

[0202] S33: To accelerate the learning process and effectively allocate credit, we introduce the eligibility trace mechanism. Based on the type of action currently selected (local, edge, or cloud), we dynamically adjust the eligibility trace's decay factor λ (stored in lambda_values), so that states on different decision paths receive differentiated learning weights.

[0203] S34: Experience tuple generated by each interaction with the environment (s t ,a t ,r t ,s t+1 ,d t , where d tIndicates whether the task is completed) is stored in an experience replay buffer (PrioritizedExperienceReplay). In particular, if the experience is generated under a simulated adversarial event (see step S4 for details), the experience is given a higher storage priority, for example, by repeatedly storing it in the buffer or assigning it a higher sampling weight, thereby increasing its probability of being sampled in subsequent learning.

[0204] S35: Periodically extract a batch of experience data from the experience playback buffer. j ,a j ,r j ,s j ′,d j ),have:

[0205] S351, calculate the next state s j ′, the expected Q value that can be obtained by following the current ε-greedy strategy:

[0206]

[0207] Sum all possible actions a, where π(a|s {j+1,d} ) is in state s {j+1,d} The probability of choosing action a.

[0208] S352. Calculate the time difference (TD) error:

[0209]

[0210] Where γ is the discount factor, s{ j,d} is state s j Discrete representation of .

[0211] S353, update the qualification trace table E(s d ,a): For all state-action pairs (s d ,a)Execute:

[0212] E(s d ,a)=γ·λ·E(s d ,a)

[0213] The state-action pair corresponding to the current experience (s {j,d} ,a j )implement:

[0214] E(s {j,d} ,a j )=E(s {j,d} ,a j )+1

[0215] S354, update Q value table Q(s d ,a): For all state-action pairs (s d ,a), execute:

[0216] Q(s d ,a)=Q(s d ,a)+α·δ·E(s d ,a)

[0217] Where α is the learning rate.

[0218] S36: To promote load balancing and knowledge sharing among edge servers, a smoothing operation is performed on the Q values ​​of the actions related to offloading to different edge servers in the Q value table at every fixed update period to converge to the mean.

[0219] Adversarial training module for realistic perturbations:

[0220] In this embodiment, the adversarial training module for real-world disturbances executes step S4. Step S4 aims to improve the robustness of the agent in dealing with unexpected situations such as network fluctuations and server failures in actual deployment. The adversarial event simulation mechanism is introduced during the training phase and specifically includes the following sub-steps:

[0221] S41: Based on typical failures or interferences that a MEC system may encounter, define specific adversarial event types:

[0222] Sudden high load / downtime of edge servers: In the simulation environment, one or more edge servers are randomly selected with a certain probability adv_prob, and their available computing resources edge_capacity are instantly reduced to an extremely low level or completely unavailable (equivalent to temporarily setting it to 0 or a very minimum value).

[0223] Sudden change in task urgency: The deadline of the simulated task, task_deadline, is suddenly significantly advanced.

[0224] S42: During the environmental interaction process of step S1, the above-defined adversarial event is triggered according to the set probability adv_prob (the probability can be set higher in the early stage of training and gradually reduced in the later stage), and the experience data generated under the influence of the event are specially marked.

[0225] S43: Through the prioritized experience replay mechanism described in step S3, ensure that the intelligent agent can learn from these simulated "crisis" scenarios more frequently, thereby adjusting its decision-making strategy to better cope with uncertainties in the real world.

[0226] By implementing steps S1 to S4 above, the present invention can train an intelligent management model that has dynamic adaptability, can autonomously make task offloading decisions in a complex and changing MEC environment, and strives to achieve optimal overall performance in multiple dimensions such as latency, energy consumption, and load balancing. After the model is trained, its core decision-making logic (i.e., the Q-value table and related parameters) can be solidified and deployed in the actual MEC control system to guide real-time task scheduling.

[0227] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

[0228] Example 3

[0229] This embodiment is based on a delay-sensitive task offloading scenario, and takes the collaborative emergency braking warning as an example to explain the aforementioned method in detail:

[0230] 1. Task triggering and state representation construction:

[0231] Task Background: A vehicle is traveling in complex traffic flow. Sensor systems (such as millimeter-wave radar and vehicle-to-everything (V2X) communication units) detect a potential rear-end collision or intersection collision ahead. The system generates a high-priority, latency-critical computational task. This task may involve multi-sensor data fusion, target trajectory prediction, and collaborative avoidance strategy generation. It has a moderate computational load (task_size) but a strict deadline (task_deadline).

[0232] Environmental state acquisition and encoding: The vehicle's decision-making agent starts the state perception module.

[0233] Task attribute vector: encodes the computational requirements of the task (task_size), the deadline (task_deadline), and the urgency flag (is_urgent).

[0234] Spatial context awareness: Based on the vehicle's current geographic location (vehicle_pos) and the deployment locations of each edge server (server_positions), the spatial attention mechanism (_spatial_attention) generates affinity weights for each edge server. The closer the distance, the higher the weight.

[0235] Dynamic network load assessment: The instantaneous load of each edge server (edge_loads) and the remote cloud load (cloud_load) are collected. Historical load data (load_history) within a sliding window (history_window) is then combined with temporal attention (_temporal_attention, which gives higher weight to recent data) and spatial attention weights to generate aggregated features of edge network load trends. The load variance (load_variance) of all computing nodes (edge ​​+ cloud) is also calculated as a measure of network balance.

[0236] 2. Offloading decision making and execution:

[0237] Policy Inference: The agent queries its learned state-action value function based on the discretized representation of the current state. According to the ε-greedy strategy, it chooses the action with the highest Q value with high probability, or explores with ε probability.

[0238] Action space analysis:

[0239] Local execution (action = 0): Suitable for situations with minimal computational effort or poor network conditions. The latency is primarily computational, and energy consumption is high.

[0240] Edge offloading (1 <= action <= 3): The agent evaluates each edge server. The key consideration is the expected total latency, which includes transmission latency and edge processing latency. If the target edge server is overloaded with the current task, a penalty of high latency will be incurred.

[0241] Cloud offloading (action = 4): Although cloud computing capabilities are powerful, the inherent high network transmission latency makes it less attractive in extremely latency-sensitive scenarios.

[0242] Reward-driven: Because this task is urgent, the reward function places a high weight on the deadline compliance of such tasks. If the deadline is exceeded, a significant negative reward is applied. This forces the agent to prioritize offloading targets that ensure completion within the task_deadline.

[0243] Adversarial perturbation injection: This simulates real-world network uncertainties, such as sudden increases in edge server load or unexpectedly compressed task deadlines. These experiences are stored in the Prioritized Experience Replay pool with a higher priority (adversarial_priority), strengthening the agent's adaptability to adverse environments.

[0244] 3. Learning and strategy iteration:

[0245] Based on the immediate reward (reward) and new state (next_state) obtained after executing an action, the agent uses the expected SARSA update rule, combined with a dynamically adjusted eligibility trace decay factor λ, to update the Q table and eligibility trace table (e_table). The prioritized experience replay mechanism ensures sufficient learning of critical (especially adversarial) examples.

[0246] For latency-sensitive tasks, the vehicle agent's offloading decisions strongly favor minimizing end-to-end latency. Even if this means choosing a slightly more computationally expensive or energy-intensive approach (such as selecting an edge server with a very low load but a poor communication link, or performing local processing when the task is very small and the network is congested), as long as the stringent latency constraints are met, this strategy will be considered the preferred one.

[0247] Example 4

[0248] This embodiment is based on a resource-optimized task offloading scenario, and takes the synchronization of in-vehicle infotainment system content as an example to explain the aforementioned method in detail:

[0249] 1. Task triggering and state representation construction:

[0250] Task Background: Vehicle passengers request to download large media files (such as HD map updates, TV series, or TV series) or perform non-real-time batch data analysis. These tasks are typically computationally intensive but have a high latency tolerance, meaning that the task_deadline is relatively relaxed.

[0251] Environmental state acquisition and encoding: Similar to scenario 1, the agent constructs a comprehensive state vector that includes task attributes, spatial context, and network load dynamics. However, the weighting of these state factors will be different.

[0252] 2. Offloading decision making and execution:

[0253] Policy inference: The process is the same as the scenario in Example 3, but the learning of Q values ​​and the final decision will be affected by different reward function components.

[0254] Action Space Parsing and Preferences:

[0255] Local execution: Since the task can be computationally intensive, local processing will result in increased vehicle energy consumption. This is heavily negatively weighted in the reward function, so the agent will tend to avoid this option.

[0256] Edge offloading becomes an attractive option. The agent prioritizes edge servers with lower resource utilization. Due to its high latency tolerance, the agent may choose an edge node that is slightly farther away but has less resources.

[0257] Cloud offloading: For computationally intensive and non-latency-sensitive tasks, the cloud becomes a viable option due to its powerful computing capabilities and potential economies of scale. Although transmission latency is inherently high, if cloud resources are abundant and the tasks are not latency-sensitive, this can help balance the load across the MEC system.

[0258] Reward-driven: When the task is routine, the reward function prioritizes efficient utilization of computing resources and balanced system load. At the same time, the penalty for energy consumption is relatively high. This guides the agent to make decisions aimed at reducing operating costs and increasing overall network throughput.

[0259] Collaborative Q-value updating: Periodically, the agent smoothes the action values ​​in its Q-table for different edge servers. This encourages the agent to have better generalization ability when selecting edge servers, avoids over-reliance on historical optimality and causes overload of certain servers, and promotes collaboration and load balancing among edge resources.

[0260] 3. Learning and strategy iteration:

[0261] The agent updates its Q function through experience gained from interacting with the environment. For non-urgent tasks, the learning process emphasizes strategies that can bring long-term resource efficiency and energy savings.

[0262] For resource-optimized tasks, the vehicle agent's offloading decisions prioritize maximizing resource utilization and minimizing energy consumption. It may choose a remote server (edge ​​or cloud) with slightly higher communication overhead but more abundant or cheaper computing resources to achieve cost-effective processing of computing tasks and promote load balancing across the MEC network. It has a high tolerance for latency and prioritizes overall system operational efficiency.

[0263] A comparative analysis of the two scenarios in Examples 3 and 4 demonstrates that the vehicle agent within this code framework can adaptively adjust its task offloading strategy based on the heterogeneous requirements of tasks (latency sensitivity, computational intensity) and the dynamically changing network environment (server load, topological distance), demonstrating intelligent decision-making capabilities in complex in-vehicle MEC environments. Mechanisms such as adversarial event injection and prioritized experience replay further enhance the robustness and efficiency of its learning.

[0264] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A multi-target offloading method for edge computing in Internet of Vehicles (IoV) that integrates dynamic environment modeling and improved SARSA, characterized by: The method comprises: Obtain dynamic vehicle network environment parameters and perform spatiotemporal perception multi-dimensional state encoding based on real-time data; Establish an adaptive strategy decision model based on improved SARSA, and perform strategy generation and optimization; The offloading action is actually executed according to the generated execution strategy, and a multi-dimensional constraint compound reward evaluation is performed based on the actual execution results; The model is optimized offline and continuously learned based on real-world disturbance perception, key experience, hierarchical credit allocation, and collaborative Q-table iterative optimization.

2. The multi-target offloading method for edge computing in Internet of Vehicles (IoV) integrating dynamic environment modeling and improved SARSA according to claim 1 is characterized by: The IoV environment includes at least vehicles, roadside units, edge servers, and cloud servers. IoV environment parameters include the number of edge servers (max_servers), the upper limit of the computing processing capacity of each edge server node (edge_capacity), the computing processing capacity of the central cloud server (cloud_capacity), the maximum task processing delay threshold (max_delay) required for quality of service, and the geographical deployment location (server_positions) of each edge server. The length of the historical data analysis window (history_window) and the maximum number of interaction steps in a single decision or evaluation cycle (episode_length) are set. At the beginning of each decision cycle, the following information is dynamically acquired or received: the current real-time location of the vehicle user device (vehicle_pos), the amount of data for the newly arrived or currently processed computing task (task_size), and the required processing deadline (task_deadline). Based on task_deadline and the current time, the task is determined to be urgent, and the task is classified as either urgent or ordinary. Urgent tasks are marked as is_urgent = True, and ordinary tasks are marked as is_urgent = False. Monitor and obtain the current computing load status of each edge server and cloud server in real time.

3. The multi-target offloading method for edge computing in Internet of Vehicles (IoV) integrating dynamic environment modeling and improved SARSA according to claim 2 is characterized by: The process of dynamic change perception of edge offloading tasks in the Internet of Vehicles environment includes the following processes: spatial correlation analysis, time series trend analysis, spatiotemporal feature fusion and aggregation, and system load balancing measurement. During the spatial correlation analysis, the physical distance between the user device and each server is calculated based on the real-time location of the user device and the fixed location of each edge server. The load information of each server is weighted using an exponential decay function to ensure that the closer the server is to the user device, the greater its load influence in the state representation. The spatial weight calculation formula is: Where, d i is the Euclidean distance vector between the user device and each edge server, i is the index of the vehicle that currently needs to make a decision; j is the index of the edge server, and N is the total number of available edge servers; In the process of time series trend analysis, a sliding window load_history containing the load data of each server in the past history_window time steps is maintained. The historical edge server load data in the window is weighted averaged using a linearly increasing weight vector to extract the historical trend of load changes, thereby obtaining the time weight w t ; In the process of spatiotemporal feature fusion and aggregation, the spatial correlation weight and time series trend are combined to calculate the current edge server's spatially weighted average load current_feature and the historical edge server load data's temporally and spatially weighted average trend feature history_features. The calculation formula is: Where, F e is the current load vector of each edge server; During the system load balancing measurement process, the statistical variance load_variance of the current load of all edge server nodes and central cloud servers is calculated as an indicator to measure whether the load of the entire system is balanced; Finally, the task attributes task_size, task_deadline, is_urgent and the environmental features current_feature, history_features, cloud_load, load_variance are combined, and the necessary normalization or standardization is performed on each component to form a fixed-dimensional state vector S t .

4. The multi-target offloading method for edge computing in Internet of Vehicles (IoV) integrating dynamic environment modeling and improved SARSA according to claim 3 is characterized by: In the adaptive strategy decision model based on improved SARSA, the continuous or high cardinality state feature S t Discretization is performed according to the binning rule, where predefined interval boundaries are mapped to discrete state indices. Then, a Q-value table q_table and a lattice trace table e_table are initialized. The Q-value table q_table is used to store the estimated value of each discrete state-action pair, and the eligibility trace table e_table is used as an auxiliary structure for subsequent learning optimization. In the decision-making process, the agent randomly selects a feasible unloading action with an exploration rate of ε; and selects the action with the maximum Q value in the current discrete state with probability 1-ε; Finally, according to the adaptive strategy π(A t |S t ) is the current state S t Select the optimal or suboptimal unloading action A t , where the updates of the Q-value table q_table and the eligibility trace table e_table, as well as the updates of the optimization mechanism depend on the feedback obtained from the environment after the action is executed.

5. The multi-target offloading method for edge computing in Internet of Vehicles (IoV) integrating dynamic environment modeling and improved SARSA according to claim 4 is characterized by: The final confirmed uninstall action A t The system converts the task into actual instructions, which are executed on the real computing and network infrastructure through the vehicle communication module and network interface. After the actual execution is completed, the system will collect and analyze the various real performance indicators of the task execution, and evaluate the effect of the decision based on the preset multi-objective reward engine to generate a scalar reward signal R t ,in, For urgent tasks, if the actual completion delay D does not exceed its task deadline D td , then a positive reward is given according to the remaining time margin, and the calculation formula is as follows: If the timeout is exceeded, a significant negative penalty will be imposed based on the duration of the excess, calculated as follows: P ou =C2·max(D-D td ,0) Where C1 and C2 are the preset reward and penalty coefficients. If the task is processed locally on the vehicle, the actual energy consumption energy_actual is deducted from the total reward according to a certain coefficient C3. For non-urgent tasks, we encourage offloading them to servers with lower actual loads. The reward is positively correlated with the estimated or actual remaining available resource ratio of the target server after offloading. The calculation formula is: R rn =C4·(1-U) Completion on time is also rewarded, but its weight is lower than that of urgent tasks. The formula is: Where C5 is the preset non-urgent task timeliness reward coefficient; the actual energy consumption of local processing, energy_actual, is penalized with an increased penalty weight and deducted from the total reward by coefficient C6; A global load variance penalty is introduced to promote load balancing. The penalty is proportional to the load variance load_variance calculated based on the current real-time load of all servers: R balance =C7·s 2 Where C7 is the load variance penalty coefficient, σ 2 is the load variance load_variance; Considering that extreme events that may occur in the real environment may cause the reward signal to fluctuate violently, the calculated single-step reward r t impose a lower limit; The above reward / penalty components are weighted and summed by the preset weight coefficients according to the importance of each optimization goal in the actual application scenario to obtain the final composite reward signal R t .

6. The multi-target offloading method for edge computing in Internet of Vehicles (IoV) integrating dynamic environment modeling and improved SARSA according to claim 5 is characterized by: In the optimization process of the adaptive strategy decision model based on the improved SARSA, the eligibility trace mechanism is introduced to accelerate the learning process and effectively allocate credits. According to the type of the currently selected action, the attenuation factor λ of the eligibility trace is dynamically adjusted. The update rule of the eligibility trace is as follows: The experience tuple (S t ,A t ,R t+1 ,S t+1 ,done) is stored in an experience replay buffer; if the experience is generated under a simulated adversarial event, the experience is given a higher storage priority; Periodically extract a batch of experience data from the experience playback buffer, for each experience (s t ,a j ,r j ,s′ j ,d j ),have: Calculate the next state s′ j Under this circumstance, the expected Q value that can be obtained by following the current ε-greedy strategy is: Sum all possible actions a, where π(a|s {j+1,d} ) is in state s {j+1,d} The probability of choosing action a; Calculate the time difference error: Where γ is the discount factor, s {j,d} For state s j Discretization representation of ; Update the qualification trace table E(s d ,a): For all state-action pairs (s d ,a)Execute: E(s d ,a)=γ·λ·E(s d ,a) The state-action pair corresponding to the current experience (s {j,d} ,a j )implement: And(s) {j,d} ,the j )=And(s) {j,d} ,the j )+1 S354, update Q value table Q(s d ,a): For all state-action pairs (s d ,a), execute: Q(s d ,a)=Q(s d ,a)+α·δ·E(s d ,a) Where α is the learning rate. Finally, every fixed update period, a smoothing operation is performed on the Q-values ​​of the actions related to offloading to different edge servers in the Q-value table to converge to the mean.

7. The multi-target offloading method for edge computing in Internet of Vehicles (IoV) integrating dynamic environment modeling and improved SARSA according to claim 6 is characterized by: An adversarial event simulation mechanism is introduced during the training phase, and simulated adversarial scenarios are used to enhance the offline or semi-online training phase. The adversarial event types are defined as: Sudden high load / downtime of edge servers: In the simulation environment, one or more edge servers are randomly selected with a certain probability adv_prob, and their available computing resources edge_capacity are instantly reduced to a very low level or completely unavailable; Sudden change in task urgency: The deadline of the simulated task, task_deadline, is suddenly brought forward; Then, based on the recorded real disturbance pattern or the preset typical failure scenario, a simulated adversarial stress test event is injected with a certain probability adv_prob; After perceiving real sudden interference or injecting simulated adversarial events during training, the resulting states, actions, rewards, next states, and completion flags will adopt special storage and utilization strategies; If the generation of an experience is associated with a confrontational event type, then this key / experience will be given a higher learning priority in the experience replay pool; In the process of sampling experience from the experience replay pool and updating the Q-table, an advanced Q-table iteration mechanism is applied to further optimize credit allocation and accelerate knowledge sharing: First, through a hierarchical credit allocation mechanism, which dynamically selects different eligibility trace decay coefficients λ based on the action type, we ensure that the obtained real reward signal can be traced back to the historical decisions that contributed to the result; Secondly, through the collaborative Q-table iteration mechanism, every preset update interval, the Q-values ​​of all edge server offloading-related actions in the Q-value table are subjected to mean regression based on the current Q-table state: the specific operations are: Among them, w is the regression weight, is the mean Q value of all edge offloading actions in the current state; Through the combined effect of the above mechanisms, the speed at which Q-value learning can adapt to the dynamics of the real environment and the performance of the final strategy in practical applications can be improved.

8. A system for executing the multi-target offloading method for edge computing in an Internet of Vehicles (IoV) integrating dynamic environment modeling and improved SARSA as described in any one of claims 1 to 7, characterized in that: The system includes a MEC environment perception and multi-dimensional state construction module, a dynamic reward feedback mechanism module for multi-dimensional performance indicators, an intelligent decision model construction and learning module based on the improved SARSA, and an adversarial training module for real-world disturbances. The MEC environment perception and multi-dimensional state construction module is used to fully perceive the edge computing environment of the Internet of Vehicles and construct a multi-dimensional state vector for reinforcement learning decision-making. It collects vehicle location, task attributes, and load information of edge servers and cloud servers in real time, and combines spatial and temporal attention mechanisms to encode the environmental state. It generates a multi-dimensional state tuple that fully reflects the current environmental conditions, providing a basis for subsequent intelligent decision-making. The dynamic reward feedback mechanism module for multi-dimensional performance indicators designs a dynamic reward function that can reflect multi-dimensional performance goals. It provides positive or negative numerical feedback to the agent based on the actual results of each task offloading decision. It dynamically calculates and generates reward signals based on task processing delay, system energy consumption, resource utilization, and load balancing to guide the agent to learn offloading strategies that meet actual application needs. The intelligent decision-making model construction and learning module based on the improved SARSA is used to build and optimize the task offloading decision model based on the enhanced state-action-reward-state-action reinforcement learning algorithm. By introducing hierarchical eligibility trace decay, collaborative Q-table updates, and expected SARSA update rules, it reduces the Q-value update variance, improves the algorithm convergence speed and long-term reward propagation efficiency, and enhances distributed policy consistency. The adversarial training module for real-world disturbances is used to improve the robustness of intelligent agents in responding to emergencies in actual deployments. By simulating adversarial event scenarios and combining experience replay priority strategies with policy entropy regularization, the intelligent agent is trained to enhance its ability to learn and adapt to abnormal scenarios and reduce the risks of policy collapse and overfitting.

9. A computer-readable storage medium, characterized in that The storage medium stores a computer program. When the computer program is executed by the processor, the multi-target offloading method for edge computing of the Internet of Vehicles that integrates dynamic environment modeling and improved SARSA as described in any one of claims 1 to 7 is implemented.

10. A computer program product, characterized in that: The invention comprises a computer program which, when executed by a processor, implements the multi-target offloading method for edge computing of the Internet of Vehicles by integrating dynamic environment modeling and improved SARSA as described in any one of claims 1 to 7.

Images