Vehicle network multi-dimensional resource flexible scheduling method based on hierarchical reinforcement learning

Abstract

The invention provides a vehicle network multi-dimensional resource flexible scheduling method based on hierarchical reinforcement learning. The method comprises the following steps: modeling each target CS in a target area of a vehicle network system as an intelligent agent; determining an optimization problem of power resource and computing power resource scheduling in a CS-ICV-user side interaction process in the target area, constructing a target function by taking the maximization of the total revenue of all agents in a scheduling time period as a target, and determining constraint conditions of the target function; a multi-agent model is constructed, the multi-agent model comprises an upper layer and a lower layer, the upper layer is used for solving the optimization problem of power resource scheduling, and the lower layer is used for solving the optimization problem of computing power resource scheduling; and solving the target function under constraint conditions by using a multi-agent model based on the observation state of each agent in the scheduling time period to obtain an optimal scheduling strategy. According to the invention, the flexibility of multi-dimensional resource scheduling is improved, and reasonable allocation of multi-dimensional resources can be realized in a complex scene.

Description

Technical Field

[0001] The present invention relates to the technical field of resource allocation, and in particular to a vehicle network multi-dimensional resource elastic scheduling method based on hierarchical reinforcement learning. Background Art

[0002] Due to the rapid development of emerging energy and information integration technologies, the application scope of Intelligent Connected Vehicles (ICV) has been significantly expanded recently. By charging or discharging to the charging station (CS) in the smart grid, ICV can provide power auxiliary services, such as peak load shaving services. In addition, ICV can also serve as an edge computing node to provide computing services to the user side during peak load shaving. At present, most existing technologies focus on a single resource bundle, which makes it difficult to achieve the reasonable allocation of multi-dimensional resources in complex scenarios. Summary of the invention

[0003] The present invention provides a vehicle network multi-dimensional resource elastic scheduling method based on hierarchical reinforcement learning, which is used to solve the defect that the prior art only focuses on a single resource bundle and is difficult to achieve reasonable allocation of multi-dimensional resources in complex scenarios.

[0004] The present invention provides a vehicle network multi-dimensional resource elastic scheduling method based on hierarchical reinforcement learning, comprising: Applied to a vehicle network system, the vehicle network system includes multiple charging stations CS, multiple intelligent networked vehicles ICV and multiple user terminals, the method includes: Each target CS in the target area of ​​the vehicle network system is modeled as an intelligent agent, and the intelligent agent obtains benefits by scheduling the power resources and computing power resources of the target ICV in the target area within the scheduling period; Determine the optimization problem of power resource and computing resource scheduling in the CS-ICV-user interaction process in the target area, take maximizing the total benefits of all agents in the scheduling time period as the goal, construct the objective function, and determine the constraints of the objective function; Based on the optimization problem modeling, a partially observable Markov decision process is constructed to construct a multi-agent model, wherein the multi-agent model includes an upper layer and a lower layer, wherein the upper layer is used to solve the optimization problem of power resource scheduling, and the lower layer is used to solve the optimization problem of computing resource scheduling; Determine the observed state of each agent in the scheduling time period, and solve the objective function under the constraint conditions based on the observed state of each agent in the scheduling time period using the multi-agent model to obtain an optimal scheduling strategy; The multi-agent model is trained based on the sample states of the sample agents and the optimal action labels of the sample agents.

[0005] In some embodiments, constructing an objective function based on the optimization problem includes: Based on the optimization problem of power resource scheduling, constructing a power peak load utility function; Based on the optimization problem of computing resource scheduling, construct a computing offloading utility function; The objective function is constructed based on the power peak load regulation utility function and the computational load shedding utility function.

[0006] In some embodiments, constructing a power peak load utility function includes: Based on the reward obtained by the target CS for completing the peak-shaving task, the peak-shaving fee that the target CS needs to pay to the target ICV, and the penalty cost incurred by the target CS for failing to complete the peak-shaving task, the power peak-shaving utility function of the target CS is constructed; The constructing of the computation offloading utility function comprises: Based on the reward obtained by the target CS for completing the computing task, the computing fee that the target CS needs to pay to the target ICV, and the penalty cost incurred by the target CS for not completing the computing task, the power peak-shaving utility function of the target CS is constructed.

[0007] In some embodiments, the objective function is calculated as follows: ; in, represents the objective function, Indicates the CS-ICV-user end matching relationship. Indicates the computing power value that ICV needs to provide. Indicates the charging or discharging power that the ICV needs to provide. Indicates Peak load scheduling cycle, Indicates the total number of peak-load scheduling cycles within the target scheduling period. Indicates The first peak load scheduling cycle Computation scheduling cycle, Indicates CS, Indicates the total number of CS in the target area, Indicates ICV, Indicates the total number of ICVs in the target area. Indicates A user terminal, Indicates the total number of users in the target area. Indicated in The peak load scheduling cycle CS and The matching relationship between ICVs, Indicated in The peak load scheduling cycle CS and The matching relationship of each user end, Indicated in In the peak load scheduling cycle The calculation scheduling cycle ICV and The matching relationship of each user end, Indicated in The peak load scheduling cycle CS calls The total revenue obtained by each ICV from completing peak load regulation and user-side computing tasks; Indicated in The peak load scheduling cycle CS calls The revenue obtained by ICVs from peak load shifting is Indicated in In the peak load scheduling cycle The calculation scheduling cycle CS calls ICV completed the The profit obtained from the computing task of each user.

[0008] In some embodiments, the constraints include: in any peak-shaving scheduling cycle, the number of target ICVs participating in power peak-shaving shall not exceed the number of available target CSs; in any peak-shaving scheduling cycle, each target ICV can only select at most one target CS to participate in power peak-shaving; in any computing scheduling cycle in any peak-shaving scheduling cycle, the computing task of each user terminal can only be assigned to at most one target ICV; each target ICV must reach the corresponding target CS within a preset time threshold; each target ICV needs to have sufficient power to reach the corresponding target CS; in the peak shaving stage, each target ICV needs to have sufficient power supply capacity after reaching the corresponding target CS; in the valley filling stage, each target ICV needs to have sufficient charging gap after reaching the corresponding target CS; the charging or discharging power provided by each target ICV is within the preset power threshold; the computing power value provided by each target ICV is within the preset computing power threshold.

[0009] In some embodiments, the parameters of the agent include observation state, action and reward; the observation state includes upper-level state and lower-level state, the upper-level state refers to the observation state of the target CS during the peak-shaving scheduling cycle, and the lower-level state refers to the observation state of the target CS during the computing scheduling cycle in the peak-shaving scheduling cycle; the action includes upper-level action and lower-level action, the upper-level action refers to the power peak-shaving action taken by the target CS during the peak-shaving scheduling cycle, and the lower-level action refers to the computing scheduling action taken by the target CS during the computing scheduling cycle; the reward includes upper-level reward and lower-level reward, the lower-level reward refers to the computing unloading benefit of the target CS during the peak-shaving scheduling cycle, and the upper-level reward refers to the power peak-shaving benefit and computing unloading benefit of the target CS during the peak-shaving scheduling cycle.

[0010] In some embodiments, the upper-level status includes: the peak-shaving capability of the target ICV, the computing capability of the target ICV, the driving distance from the target ICV to the target CS, the type of computing tasks that the target ICV is good at, peak-shaving instructions, the number of charging piles of the target CS, the type of the current peak-shaving phase, the price of unit electricity involved in peak-shaving, the matching relationship between the target user terminal and the target CS, and the basic information of the computing tasks of the target user terminal; the lower-level status includes: the peak-shaving capability of the target ICV, the computing capability of the target ICV, the upper-level action, the type of computing tasks that the target ICV is good at, the Poisson distribution parameters of the computing tasks, the matching relationship between the target user terminal and the target CS, and the basic information of the computing tasks of the target user terminal.

[0011] In some embodiments, the upper-layer action includes the matching relationship between the target ICV and the target CS, and the charging or discharging power that the target ICV needs to provide to participate in peak regulation; the lower-layer action includes: the matching relationship between the target ICV and the target user terminal, and the computing power value that the target ICV needs to provide to participate in computing tasks.

[0012] In some embodiments, the CS-ICV-user terminal interaction process includes: During the peak-shaving scheduling period, each target CS selects a target upper layer action according to the current upper layer state of each target CS, and the target ICV matching each target CS moves to each target CS for charging or discharging; In the computing scheduling period in the peak-shaving scheduling period, each target CS selects a target lower-layer action according to the current lower-layer state of each target CS, and the target ICV matching each target CS completes the computing task of the target user end.

[0013] In some embodiments, the training process of the multi-agent model includes: Initializing the multi-agent model; Acquire an upper layer local observation state, and select an upper layer action according to the upper layer local observation state; Execute the above-mentioned upper-level actions and obtain upper-level rewards; Initialize the matching relationship between the lower-level global state and CS, ICV, and user end; According to the current global state of the lower layer, obtain the local observation state of the lower layer; Selecting a lower-layer action according to the current lower-layer global state and the matching relationship between the current CS, ICV, and user terminal; Execute the lower-level action, obtain the lower-level reward, the next lower-level global state, and the end signal; Store experience in the lower experience pool; Randomly extract data samples from the lower experience pool, train the lower Critic network and the lower Actor network, and update the lower target network every n steps, where n is a natural number greater than or equal to 1; Update upper-level rewards and store new experience in the upper-level experience pool; Data samples are randomly extracted from the upper experience pool, the upper Critic network and the upper Actor network are trained, and the upper target network is updated every n steps.

[0014] The vehicle-grid multi-dimensional resource elastic scheduling method based on hierarchical reinforcement learning provided by the present invention models each target CS in the target area of ​​the vehicle-grid system as an intelligent agent, takes maximizing the total benefit of all agents in the scheduling time period as the goal, constructs an objective function, determines the constraints of the objective function, and constructs a multi-agent model. The multi-agent model includes an upper layer and a lower layer. The objective function is solved under the constraints using the multi-agent model to obtain the optimal scheduling strategy, thereby improving the flexibility of the coordinated scheduling of power resources and computing power resources, and being able to achieve reasonable allocation of multi-dimensional resources in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 is a schematic diagram of the structure of a vehicle network system provided by an embodiment of the present invention; Figure 2 It is a flow chart of a vehicle-network multi-dimensional resource elastic scheduling method based on hierarchical reinforcement learning provided by an embodiment of the present invention; Figure 3is a schematic diagram of the structure of a multi-agent model provided by an embodiment of the present invention; Figure 4 is a comparison chart of training results of different models provided by an embodiment of the present invention; Figure 5 is a schematic diagram of peak load shaving rewards under different Gaussian distributions provided by an embodiment of the present invention; Figure 6 is a schematic diagram of reward calculation under different Poisson distributions provided by an embodiment of the present invention; Figure 7 It is a schematic diagram of benefits at different stages provided by an embodiment of the present invention; Figure 8 is a schematic diagram of the matching relationship between a charging station and an intelligent network-connected vehicle provided by an embodiment of the present invention; Fig. 9 is a schematic diagram of the relationship between peak load demand and power supply provided by an embodiment of the present invention; Fig.10 is a schematic diagram of the completion of a computing task provided by an embodiment of the present invention; Fig.11 is a schematic diagram of scalability experimental results provided by an embodiment of the present invention; Fig.12 It is a schematic diagram of ablation experiment results provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] Figure 1 The structure diagram of the vehicle network system provided by the embodiment of the present invention is shown in FIG. Figure 1 As shown, a vehicle network system is provided, which includes multiple charging stations CS, multiple intelligent connected vehicles ICV and multiple user terminals, and the multiple CSs are communicatively connected with the multiple ICVs and the multiple user terminals.

[0019] Optionally, the vehicle grid system further includes a power grid, which is communicatively connected to the plurality of charging stations CS, and the power grid is used to send peak load shaving instructions to the plurality of charging stations CS.

[0020] It should be noted that multiple user terminals continuously generate different types of computing tasks according to a certain distribution. Each ICV can provide electricity and computing power resources. Each CS is used to receive peak-shaving instructions sent by the power grid and dispatch multiple ICVs in the target area to participate in peak-shaving services. In addition, each CS is also responsible for aggregating the computing tasks of the user terminals and offloading them to multiple ICVs participating in peak-shaving.

[0021] Optionally, the peak load regulation process is divided into two modes: peak load shaving and valley load filling; certain periods of the day are set as scheduling cycles , and further divide it into a series of equal-length time slots ;Will is defined as the length of a time slot and assumes ,in For the peak shaving stage, It is the valley filling stage; social distance is used to measure the closeness between two subjects.

[0022] Optionally, the vehicle network system operates according to the following workflow: 1) The power grid issues peak-shaving instructions to multiple CSs, and at the same time, the user side generates different computing tasks, which are assigned to different CSs according to their social distance, update the system status and start the peak-shaving period; 2) Each CS publishes a peak-shaving contract to attract ICVs to participate in peak-shaving and deal with competition issues. ICVs evaluate these contracts based on their preferences and social distance, thereby updating the system status and entering the calculation cycle; 3) During the peak load period, CS provides computing contracts to its internal ICVs to encourage them to complete computing tasks on the user side. This process will last for multiple computing cycles until the current peak load period ends and the next peak load period begins.

[0023] It should be noted that each CS is concerned with the total revenue generated during the peak-shaving period. The revenue of each CS is a combination of the peak-shaving business revenue and the computing task revenue. The overall goal of the system is to enable all CS to obtain maximum benefits during the entire peak-shaving cycle of a day. By optimizing the allocation of tasks, computing resources, and peak-shaving activities, the system aims to improve the revenue-generating ability of each CS and maximize the overall revenue of the system during the peak-shaving period.

[0024] Figure 2 The flowchart of the vehicle network multi-dimensional resource elastic scheduling method based on hierarchical reinforcement learning provided by the embodiment of the present invention is as follows. Figure 2 As shown in the figure, a vehicle network multi-dimensional resource elastic scheduling method based on hierarchical reinforcement learning is provided, and the vehicle network multi-dimensional resource elastic scheduling method based on hierarchical reinforcement learning is applied to Figure 1The vehicle network system in FIG. 2 is used as an example to illustrate the method, which includes the following steps: step 210, step 220, step 230 and step 240. The method flow steps are only a possible implementation mode of the present invention.

[0025] Step 210: Model each target CS in the target area of ​​the vehicle network system as an intelligent agent, and the intelligent agent obtains revenue by scheduling the power resources and computing power resources of the target ICV in the target area within the scheduling period.

[0026] Step 220: determine the optimization problem of scheduling power resources and computing power resources during the CS-ICV-user interaction in the target area, take maximizing the total benefits of all agents in the scheduling time period as the goal, construct an objective function, and determine the constraints of the objective function.

[0027] Optionally, a CS model, an ICV model and a user-side model are constructed.

[0028] Optionally, region Inside CS is represented by , No. The position of the CS is , No. The number of charging piles for a CS is , the charging or discharging power provided by CS is ,use or Respectively, the set of CS is represented by express.

[0029] It should be noted that CS involves two functions: valley filling and peak shaving. "Valley filling" refers to the strategic charging of ICV batteries during off-peak hours when ICV owners do not need to use the vehicle. This practice helps to utilize excess power generation capacity during low-demand periods. Valley filling is usually carried out when electricity prices are low. "Peak shaving" refers to arranging ICVs to discharge to CS that meets specific standards during peak electricity consumption. This method helps to alleviate grid pressure during high-demand periods.

[0030] Alternatively, consider A collection of ICVs , a certain car The battery capacity is (kWh), the current state of charge (SoC) is The current location is determined by longitude and latitude Indicates that charging stations and cars The distance between It represents the estimated driving distance of the vehicle’s planned route; the ICV in CS has the ability to provide computing services to nearby users with computing needs and gain benefits from them; it defines the ICV in the area The position distribution within is , Indicates car CPU cycle frequency (Cycles / Second), used to measure the car computing power, cars The types of computing tasks that it excels at are also different. Indicates car A collection of computing task types that it excels at.

[0031] Charging Station and cars The actual distance is expressed as , Indicates car The average moving speed of cars Overcoming friction and air resistance to move to the charging station The energy consumption is defined as: ; in , , , , and Represents cars mass, gravitational acceleration, ground friction coefficient, speed, air resistance and frontal area.

[0032] It should be noted that, considering a set of User collection , each client Each of them has a computing task to be processed. Usually, ICV can perform computing tasks such as power prediction, route planning, traffic data analysis and image processing. The computing task is represented by a four-tuple. , Indicates the size of the task data (bytes). Indicates the task type. represents the delay limit of the task, is the reward per byte for completing the task, in units of $ / byte; in order to ensure a reasonable computing time and avoid excessive delays, it is assumed that the sizes of different tasks are evenly distributed, and the sizes of different tasks will not differ by orders of magnitude to avoid the computing time being too long or too short. In addition, the generation of computing tasks follows a certain distribution , the number of CPU cycles required to calculate each input bit is fixed, and the calculation density is defined as (Cycles / byte).

[0033] In some embodiments, constructing an objective function based on an optimization problem includes: Based on the optimization problem of power resource scheduling, the power peak load utility function is constructed; Based on the optimization problem of computing resource scheduling, a computing offloading utility function is constructed; Based on the power peak load regulation utility function and the computational load shedding utility function, the objective function is constructed.

[0034] Optionally, an ICV-CS interaction model and an ICV-user-end interaction model are constructed.

[0035] It should be noted that the architecture of ICV and CS revolves around CS issuing peak load-shaving instructions to nearby ICVs. Whether ICV chooses to participate in peak load-shaving depends on factors such as the distance from CS and whether there are idle chargers. Assuming that the next scheduling cycle Recruitment of ICV from the previous scheduling cycle In addition, ICV needs to be completed at the required time The estimated time for the i-th ICV to arrive at the charging station is ,in .

[0036] It should be noted that during the valley filling period, when the power demand of the grid is low, a peak load regulation instruction will be issued to require ICVs to charge. It is necessary to recruit some ICVs with relatively low battery power for charging. When ICV Receive recruitment and go to CS To charge, there must first be enough energy to reach the charging station. Secondly, the ICV needs to continue charging during the scheduling cycle to complete the specified peak load instruction. for: ; in, ICV The state of charge when receiving the peak load instruction, is the scheduling cycle duration, is the charging power, is the maximum state of charge of the ICV, ICV of battery capacity.

[0037] It should be noted that when the power grid has a peak in electricity consumption, the peak load command issued requires ICVs to discharge to the grid, which requires recruiting some ICVs with higher battery power to discharge. When ICV Accept the recruitment and go to the charging station The ICV needs to discharge continuously during the scheduling cycle to complete the remaining power after the specified peak load instruction is completed. for: ; in ICV The discharge power, ICV safe state of charge.

[0038] It should be noted that during the peak load period, CS selects specific computing tasks from many user terminals and then matches them to ICVs participating in the peak process. During the peak load period, ICVs can accept users' computing offload tasks based on their own computing capabilities and preferences, and earn revenue by completing these tasks. Since ICVs have limited computing capabilities for large computing tasks, each ICV has a certain number of computing tasks in the scheduling cycle. At most one computing task can be selected within the network, and all of them will be offloaded to the edge for processing.

[0039] Optionally, ICV Handling Users The time required for the computational task The calculation formula is as follows: ; in, ICV The compute capability values ​​available.

[0040] Optionally, the process of offloading computing tasks from the user end to the ICV is carried out on the wireless channel. According to Shannon's theorem, the user end ICV Maximum uplink transmission rate for offloaded computing tasks It is expressed as follows: ; in, represents the channel bandwidth, Represents the transmission rate of the user end. represents the wireless channel gain, represents the noise power, Represents the interference power during the unloading process.

[0041] Optionally, the delay during transmission is expressed as follows: ; Optionally, the user Offloading computational tasks to ICV Overall latency for processing It is expressed as follows: .

[0042] In some embodiments, constructing a power peak load utility function includes: Based on the reward obtained by the target CS for completing the peak-shaving task, the peak-shaving fee that the target CS needs to pay to the target ICV, and the penalty cost incurred by the target CS for failing to complete the peak-shaving task, the power peak-shaving utility function of the target CS is constructed; Construct a computation offloading utility function, including: Based on the reward obtained by the target CS for completing the computing task, the computing fee that the target CS needs to pay to the target ICV, and the penalty cost incurred by the target CS for not completing the computing task, the power peak-shaving utility function of the target CS is constructed.

[0043] It should be noted that CS publishes its basic information such as peak-shaving type, peak-shaving gap, peak-shaving compensation and number of idle charging piles, and provides a contract to ICVs willing to participate in peak-shaving, and performs power peak-shaving according to the following process: CS releases basic information: ;in Indicates peak clipping, It means filling the valley; Indicates the price of the unit electricity involved in peak load regulation. When ICV participates in peak load reduction, , when participating in filling the valley ; Compensation for the loss of ICV battery life per unit of power; It is the penalty fee for not completing peak load regulation; the above prices are in $ / kWh. is the dispatching fee per unit mileage, in $ / km; CS provides peak load contract , It is the charging or discharging power that the ICV participating in peak load regulation can provide. , when discharging , The upper and lower bounds of the value are ; ICV chooses to accept or reject the peak-shaving contract based on information such as the current location and status. If it rejects, the process will be terminated. If ICV accepts the peak-shaving contract, the peak-shaving utility of CS will be calculated based on the completion of the peak-shaving contract.

[0044] It should be noted that ICV selects CS for power peak shifting based on its state of charge, physical distance, and charging station rewards. When CS dispatches ICV to participate in peak shifting, it not only has to pay the ICV owner for the electricity used, but also needs to pay a certain mileage fee to compensate for the energy consumption of ICV going to the charging station. If CS fails to meet the peak-shaving instruction, it will incur additional penalty costs to compensate for the cost of dispatching other resources by the power grid. At the same time, the charging or discharging power provided by ICV will affect the battery life of ICV and will cause certain economic losses.

[0045] Optionally, during the peak load scheduling period The utility function of power peak load regulation in is as follows: ; ; ; ; ; ; ; in, Indicates that in the peak scheduling cycle Internal CS Scheduling ICV The benefits generated by participating in peak load regulation; Indicates that in the peak scheduling cycle Internal CS Rewards for completing peak load shaving. Indicates that in the peak scheduling cycle Internal CS Payable to ICV The cost of Indicates the price of unit electricity participating in peak load regulation, is the charging or discharging power, when ICV When charging, , when ICV is discharged, , Peak load scheduling cycle Length of time, CS ICV Compensation for battery life loss, CS Scheduling ICV Mileage compensation costs, CS Scheduling ICV The penalty cost of peak load regulation is Scheduling ICV The load provided can meet the peak load instruction, and the penalty cost , if CS Scheduling ICV If the load provided fails to meet the peak load regulation instruction, the unit price As penalty cost for unfinished parts, ICV Battery loss costs from charging or discharging, is the battery loss unit price, Indicates the maximum safe power.

[0046] It should be noted that the calculation offloading process is as follows: The user publishes its basic information ,in Indicates the size of the task data in bytes. Indicates the task type. represents the delay limit of the task, It is the reward per byte for completing the task, in $ / byte; CS provides a computing contract ,in Indicates the ICV at the CS peak Can be used by the user The computing power provided; ICV Choose to accept or reject the contract based on the content of the computing contract and its location and status information. If rejected, the process will be terminated. If accepted, the computing contract will be calculated based on the completion of the computing contract. The computing offloading utility of CS.

[0047] It should be noted that CS selects the computing tasks of the user end according to the computing power and availability of the ICVs participating in the peak load regulation at the station. and calculation scheduling cycle In this case, CS offloads the computing task to the ICV which is currently peaking. , the benefit of CS completing the computing task is defined as the difference between the benefit of the completed computing task and the penalty of the unfinished computing task. At the same time, CS also pays ICV Certain calculation reward, compensation ICV The computational cost of the task is calculated, where the computational reward value is equal to a portion of the computational task revenue.

[0048] Optionally, during the peak-load scheduling period The calculation scheduling cycle in The utility function of power peak load regulation in is as follows: ; ; ; ; in, Indicates that in the peak load scheduling cycle The calculation scheduling cycle in Internal CS Scheduling ICV Complete the user side The revenue generated by the computing tasks, is the size of the completed computing task, in bytes. is the reward per byte for completing the computation task. The penalty price for unfinished computing tasks, in $ / byte. Indicates that the user end is not completed The penalty cost of the computation task, For unsatisfied users The penalty unit price for the delay requirement is $ / min. Indicates that the user end is not satisfied The penalty cost of the delay requirement, Indicates payment to ICV Calculate remuneration, ; ICV The energy cost per CPU cycle is ICV The energy consumption per CPU cycle, ICV Preference for computing tasks in which they excel, ICV Complete the user side The energy cost of the computing task.

[0049] Optionally, the acceptance or rejection of the contract is measured using social distance, which is defined as: ; ; in, ICV With the user end social distance, ICV With CS social distance, Dimensions that represent social characteristics, such as task type, proficiency type, geographic location, etc. ICV The n-dimensional social feature vector, Indicates the user end The n-dimensional social feature vector, CS The n-dimensional social feature vector of .

[0050] Step 230: Based on the partially observable Markov decision process modeling of the optimization problem, a multi-agent model is constructed. The multi-agent model includes an upper layer and a lower layer. The upper layer is used to solve the optimization problem of power resource scheduling, and the lower layer is used to solve the optimization problem of computing power resource scheduling.

[0051] Among them, the multi-agent model is trained based on the sample states of the sample agents and the optimal action labels of the sample agents.

[0052] Optionally, based on the optimization problem, the CS-ICV-user-side interaction process is modeled as a partially observable Markov decision process.

[0053] Optionally, define a set of parameters ,in Is a group of CS, represents the global state space of the environment, Represents the state of each CS, is the joint action space of all CSs, ; For each agent , It's CS Local observations of It's CS The action space of each agent; the strategy of each agent is a probability distribution; the function Defines the Markov transition probability distribution; function defines the expected reward for a state-action pair, is take Instant rewards, Representation status The sum of the reward discounts of is the discount factor; deterministic strategy Map each state to Specific actions in Indicates the end signal.

[0054] Figure 3 The schematic diagram of the structure of the multi-agent model provided by the embodiment of the present invention is as follows. Figure 3 As shown, the multi-agent model includes an upper layer and a lower layer.

[0055] It should be noted that during the peak load scheduling cycle When CS selects the upper-level action from the current upper-level state, ICV responds based on social distance; then, in the calculation scheduling cycle When , CS selects the lower-layer action according to the current lower-layer state, and the user end responds according to the social distance; At the end, CS will receive the lower-level reward and the system will move forward to , and update the relevant information of ICV and user side accordingly; At the end, CS receives immediate rewards from the upper layer and multiple rewards from the lower layer. The system advances to the time period with the sum of rewards. , and update all status information of CS, ICV and user end accordingly.

[0056] Step 240: determine the observed state of each agent during the scheduling period, and use the multi-agent model to solve the objective function under constraints based on the observed state of each agent during the scheduling period to obtain the optimal scheduling strategy.

[0057] Optionally, the observed state of each multi-agent is input into the multi-agent model, and the multi-agent model outputs the optimal action of each multi-agent, where the optimal action includes the optimal upper-level action and the optimal lower-level action.

[0058] Optionally, the optimal scheduling strategy is determined based on the optimal upper-level action and the optimal lower-level action of each multi-agent.

[0059] In an embodiment of the present invention, each target CS in the target area of ​​the vehicle network system is modeled as an intelligent agent, with the goal of maximizing the total benefit of all agents in the scheduling time period, an objective function is constructed, the constraints of the objective function are determined, and a multi-agent model is constructed. The multi-agent model includes an upper layer and a lower layer. The multi-agent model is used to solve the objective function under the constraints to obtain the optimal scheduling strategy, thereby improving the flexibility of the coordinated scheduling of power resources and computing power resources, and being able to achieve reasonable allocation of multi-dimensional resources in complex scenarios.

[0060] In some embodiments, the objective function is calculated as follows: ; in, represents the objective function, Indicates the CS-ICV-user end matching relationship. Indicates the computing power value that ICV needs to provide. Indicates the charging or discharging power that the ICV needs to provide. Indicates Peak load scheduling cycle, Indicates the total number of peak-load scheduling cycles within the target scheduling period. Indicates The first peak load scheduling cycle Computation scheduling cycle, Indicates CS, Indicates the total number of CS in the target area, Indicates ICV, Indicates the total number of ICVs in the target area. Indicates A user terminal, Indicates the total number of users in the target area. Indicated in The peak load scheduling cycle CS and The matching relationship between ICVs, Indicated in The peak load scheduling cycle CS and The matching relationship of each user end, Indicated in In the peak load scheduling cycle The calculation scheduling cycle ICV and The matching relationship of each user end, Indicated in The peak load scheduling cycle CS calls The total revenue obtained by each ICV from completing peak load regulation and user-side computing tasks; Indicated in The peak load scheduling cycle CS calls The revenue obtained by ICVs from peak load shifting is Indicated in In the peak load scheduling cycle The calculation scheduling cycle CS calls ICV completed the The profit obtained from the computing task of each user.

[0061] In some embodiments, the constraints include: in any peak-shaving scheduling cycle, the number of target ICVs participating in power peak-shaving shall not exceed the number of available target CSs; in any peak-shaving scheduling cycle, each target ICV can only select at most one target CS to participate in power peak-shaving; in any computing scheduling cycle in any peak-shaving scheduling cycle, the computing task of each user terminal can only be assigned to at most one ICV; each target ICV must reach the corresponding target CS within a preset time threshold; each target ICV needs to have sufficient power to reach the corresponding target CS; in the peak shaving stage, each target ICV needs to have sufficient power supply capacity after reaching the corresponding target CS; in the valley filling stage, each target ICV needs to have sufficient charging gap after reaching the corresponding target CS; the charging or discharging power provided by each target ICV is within the preset power threshold; the computing power value provided by each target ICV is within the preset computing power threshold.

[0062] Optionally, the constraint expression is as follows: ; ; ; ; ; ; ; ; ; ; ; ; ; in, represents the peak shaving scheduling period, Indicates The discharge power of an ICV, represents the first power threshold, represents the valley-filling scheduling period, Indicates The charging power of an ICV, represents the second power threshold, Indicates The number of charging piles for each CS, Indicates ICV arrives at The estimated time for each CS, represents the time threshold, Indicated in The peak load scheduling cycle ICV moves to The power consumed by each CS; Indicates The charge state of the ICV when it receives the peak load command. Indicates The battery capacity of the ICV, Indicates The safe state of charge of the ICV, Indicates the maximum state of charge threshold, Indicated in The peak load scheduling cycle The charging or discharging power provided by the ICV, Indicated in The peak load scheduling cycle The minimum power threshold of the ICV, Indicated in The peak load scheduling cycle The maximum power threshold of the ICV; Indicated in In the peak load scheduling cycle The calculation scheduling cycle ICV to The computing power value provided by the client, Indicated in In the peak load scheduling cycle The calculation scheduling cycle The minimum computing power threshold for each ICV. Indicated in In the peak load scheduling cycle The calculation scheduling cycle The maximum computing power threshold of each ICV. The matching relationship is 0 or 1, 0 means no match, and 1 means match.

[0063] In some embodiments, the parameters of the intelligent agent include observation state, action and reward; the observation state includes upper-level state and lower-level state, the upper-level state refers to the observation state of the target CS during the peak-shaving scheduling cycle, and the lower-level state refers to the observation state of the target CS during the computing scheduling cycle in the peak-shaving scheduling cycle; the action includes upper-level action and lower-level action, the upper-level action refers to the power peak-shaving action taken by the target CS during the peak-shaving scheduling cycle, and the lower-level action refers to the computing scheduling action taken by the target CS during the computing scheduling cycle; the reward includes upper-level reward and lower-level reward, the lower-level reward refers to the computing unloading benefit of the target CS during the peak-shaving scheduling cycle, and the upper-level reward refers to the power peak-shaving benefit and all computing unloading benefits of the target CS during the peak-shaving scheduling cycle.

[0064] In some embodiments, the upper-level status includes: the peak-shaving capability of the target ICV, the computing capability of the target ICV, the driving distance from the target ICV to the target CS, the type of computing tasks that the target ICV is good at, peak-shaving instructions, the number of charging piles of the target CS, the type of the current peak-shaving phase, the price of unit electricity involved in peak-shaving, the matching relationship between the target user terminal and the target CS, and the basic information of the computing tasks of the target user terminal; the lower-level status includes: the peak-shaving capability of the target ICV, the computing capability of the target ICV, the upper-level action, the type of computing tasks that the target ICV is good at, the Poisson distribution parameters of the computing tasks, the matching relationship between the target user terminal and the target CS, and the basic information of the computing tasks of the target user terminal.

[0065] Optionally, the basic information of the target user-side computing task includes at least the size, type, delay limit and unit remuneration of the computing task.

[0066] In some embodiments, the upper-level actions include the matching relationship between the target ICV and the target CS, and the charging or discharging power that the target ICV needs to provide to participate in peak regulation; the lower-level actions include: the matching relationship between the target ICV and the target user terminal, and the computing power value that the target ICV needs to provide to participate in computing tasks.

[0067] In some embodiments, the CS-ICV-user interaction process includes: During the peak-shaving scheduling period, each target CS selects a target upper-layer action according to the current upper-layer state of each target CS, and the target ICV matching each target CS moves to each target CS for charging or discharging; In the computing scheduling period in the peak-shaving scheduling period, each target CS selects a target lower-layer action according to the current lower-layer state of each target CS, and the target ICV matching each target CS completes the computing task of the target user end.

[0068] Optionally, in the upper layer, ICV The state of charge SoC is updated according to the following formula: ; in, Indicates that in the peak load scheduling cycle Internal ICV state of charge.

[0069] Optionally, in the lower layer, ICV The state of charge SoC is updated according to the following formula: ; in, Indicates that in the peak load scheduling cycle The calculation scheduling cycle in The state of charge inside.

[0070] Optionally, in the upper layer, if a certain cycle ends, ICV If the requirements are still met, the ICV You can continue to participate in subsequent processes.

[0071] In some embodiments, the training process of the multi-agent model includes: Initialize the multi-agent model; Obtain the upper layer local observation state, and select the upper layer action according to the upper layer local observation state; Perform upper-level actions and obtain upper-level rewards; Initialize the matching relationship between the lower-level global state and CS, ICV, and user end; According to the current global state of the lower layer, obtain the local observation state of the lower layer; Select the lower-layer action based on the current lower-layer global state and the matching relationship between the current CS, ICV, and user end; Execute the lower-level action, obtain the lower-level reward, the next lower-level global state, and the end signal; Store experience in the lower experience pool; Randomly extract data samples from the lower experience pool, train the lower Critic network and the lower Actor network, and update the lower target network every n steps, where n is a natural number greater than or equal to 1; Update upper-level rewards and store new experience in the upper-level experience pool; Randomly extract data samples from the upper experience pool, train the upper Critic network and the upper Actor network, and update the upper target network every n steps.

[0072] Optionally, an enhanced priority replay mechanism is introduced to assign importance weights to experiences and only retain important experiences, thereby improving the efficiency and effectiveness of model training.

[0073] Experimental setup: Table 1 is the experimental parameter setting table. As shown in Table 1, four groups of experiments E1, E2, E3 and E4 are set, where the number of charging stations is 3, 5, 6 and 10 respectively, and the number of intelligent connected vehicles is 30, 50, 60 and 100 respectively. Peak load scheduling time step Set to 30 minutes to calculate the scheduling time step It is set to 10min, and the total scheduling cycle is 6 hours. These ICVs are randomly distributed in an area of ​​20km×20km. The location of the charging station is fixed in this area. The charging and discharging prices are [0.6-0.8]$ / kwh and [0.2-0.5]$ / kwh respectively. The battery loss price is 0.05$ / kWh, the penalty for unfinished peak load is 0.15$ / kWh, the mileage compensation is 0.1$ / km, and the maximum safe charging and discharging power loss is 0.05$ / kW and 0.1$ / kW. The safety power is set to 0.15. The maximum power of the ICV itself is 60kWh. It is assumed that the distribution of ICVs is Gaussian distribution, and the attribute of the computing task is Poisson distribution.

[0074] Table 1 Experimental parameter settings The multi-agent model is based on 12 networks, 6 in the upper and lower layers. Taking the upper layer as an example, the 6 networks are Actor, target Actor, Critic1, Critic2, target Critic1, and target Critic2. The Actor network uses Leaky ReLU as the activation function of the middle layer, and has two output layers, which are responsible for outputting continuous and discrete actions. It uses gumbel_softmax to output discrete actions and sigmoid to output continuous actions. The learning rates of the upper layer Actor and Critic are 0.0004 and 0.004 respectively, and the learning rates of the lower layer Actor and Critic are 0.0002 and 0.002 respectively. The model training is carried out by dynamically adjusting the learning rate. Discount factor Set to 0.99. Soft Update Parameters Set to 0.005. The experience replay pool capacity is 200-500. The batch size is set to 64, and a total of 3000 episodes are trained.

[0075] Performance Analysis: Figure 4 The following is a comparison chart of the training results of different models provided by the embodiment of the present invention. Figure 4As shown in the figure, each model was trained 3000 times. The results show that the average total CS income (calculated as the average of the final 1000 Episodes reward) obtained by the multi-agent model-based method proposed in the embodiment of the present invention is 8660.54 US dollars, and the maximum is 9395.13 US dollars. This is 22.2% and 21.9% higher than the MATD3 model, and 27.9% and 17.1% higher than the Greedy model. In addition, the method proposed in the embodiment of the present invention is better at handling the two-layer interaction relationship between energy and information sources, while Greedy, MATD3 and other traditional RL algorithms cannot capture these uncertainties and complex relationships.

[0076] Table 2 is a performance comparison table of different models. As shown in Table 2, the total benefit (index 1) is divided into peak benefit (index 2) and calculation benefit (index 3). The models of the present invention (Ours-1, Ours-2 and Ours-3) are significantly better than the comparison models. The peak matching failure rate (index 4) and the calculation matching failure rate (index 5) indicate the number of matches rejected by ICV during the peak or calculation process due to competition or social distance, excluding ICVs that did not participate in the matching process. Training time (index 6) measures the duration of training. Greedy has a very low failure rate because the strategy adopts the principle of nearest matching and there is no competition. It can be seen from the experimental data that the model of the present invention is excellent in terms of overall benefit, profit of individual components and training time. In addition, it can effectively improve performance even in the case of fierce competition.

[0077] Table 2 Performance comparison of different models Distribution has an important impact on the decision-making of the agent. Gaussian distribution reflects the spatial distribution of ICV, while Poisson distribution This affects the properties of the computing task.

[0078] Figure 5 Schematic diagram of peak load rewards under different Gaussian distributions provided by the embodiment of the present invention. Figure 5 As shown in the figure, the benefits of peak load shifting vary with the mean and variance of the Gaussian distribution. The highest reward occurs at (0.8, 0.08), when the distribution center is close to the three CSs. This shows that higher returns can be obtained when the distribution center is close to the CS and the variance is moderate; this is because when the CS places the ICV close to the CS, the compensation costs related to mileage and battery degradation can be reduced, while also enhancing social distance. Appropriate variance can slow down competition, thereby jointly increasing the likelihood of the contract being accepted.

[0079] Figure 6 Schematic diagram of reward calculation under different Poisson distributions provided by the embodiment of the present invention. Figure 6 As shown in the figure, the calculation stage returns vary with the Poisson distribution parameter The changes can be seen in There are higher yields nearby.

[0080] Figure 7 The following is a schematic diagram of the benefits at different stages provided by the embodiment of the present invention. Figure 7 Figure 2 shows the gains at different stages of a training run, highlighting the consistency and potential importance of computing the gains.

[0081] Figure 8 A schematic diagram of the matching relationship between a charging station and an intelligent network-connected vehicle provided in an embodiment of the present invention. Figure 8 Figure 3 depicts the number of ICVs selected at different times for different CSs, showing how to optimize the charging port utilization to achieve the highest efficiency.

[0082] Fig. 9 A schematic diagram of the relationship between peak load demand and power supply provided by an embodiment of the present invention. Fig. 9 As shown in the figure, during the scheduling period, the power supply of CS-1, CS-2 and CS-3 can meet the peak load demand.

[0083] Fig.10 A schematic diagram of the completion of a computing task provided by an embodiment of the present invention. As shown in the figure, most computing tasks can be completed by using the method of the present invention.

[0084] Scalability Testing: To evaluate the scalability of the proposed method, four sets of scalability experiments E1, E2, E3 and E4 are outlined.

[0085] FIG. 11 is a schematic diagram of the scalability experimental results provided by an embodiment of the present invention. Fig.11 As shown, the results of E1, E2, E3 and E4 are shown. The experimental results show that the model of the present invention has strong scalability and can provide convergent results at various scales.

[0086] Ablation experiment: The ablation experiments mainly focus on two parts: the non-stationary state of the upper layer and the interaction of rewards between the upper and lower layers. In the first ablation experiment, the enhancement of the experience pool and the network training technique were removed, while in the second ablation experiment, the interaction of rewards between the upper and lower layers was ignored.

[0087] Fig.12 Schematic diagram of ablation experiment results provided by an embodiment of the present invention. Fig.12As shown, by using the experience pool weight and maintaining a reasonable network training speed, the non-steady-state challenge can be effectively solved. The method of the present invention utilizes a newer lower-level strategy to alleviate the instability of the strategy. The double-layer structure of the multi-agent model of the present invention can successfully solve the multi-stage matching optimization problem, thereby easily adapting to various multi-stage problems and being suitable for a variety of application scenarios.

[0088] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-dimensional resource elastic scheduling method for vehicle network based on hierarchical reinforcement learning, characterized in that: Applied to a vehicle network system, the vehicle network system includes multiple charging stations CS, multiple intelligent networked vehicles ICV and multiple user terminals, the method includes: Each target CS in the target area of ​​the vehicle network system is modeled as an intelligent agent, and the intelligent agent obtains benefits by scheduling the power resources and computing power resources of the target ICV in the target area within the scheduling period; Determine the optimization problem of power resource and computing resource scheduling in the CS-ICV-user interaction process in the target area, take maximizing the total benefits of all agents in the scheduling time period as the goal, construct the objective function, and determine the constraints of the objective function; Based on the optimization problem modeling, a partially observable Markov decision process is constructed to construct a multi-agent model, wherein the multi-agent model includes an upper layer and a lower layer, wherein the upper layer is used to solve the optimization problem of power resource scheduling, and the lower layer is used to solve the optimization problem of computing resource scheduling; Determine the observed state of each agent in the scheduling time period, and solve the objective function under the constraint conditions based on the observed state of each agent in the scheduling time period using the multi-agent model to obtain an optimal scheduling strategy; The multi-agent model is trained based on the sample states of the sample agents and the optimal action labels of the sample agents.

2. The vehicle-network multi-dimensional resource elastic scheduling method based on hierarchical reinforcement learning according to claim 1 is characterized in that: The constructing of an objective function based on the optimization problem comprises: Based on the optimization problem of power resource scheduling, constructing a power peak load utility function; Based on the optimization problem of computing resource scheduling, construct a computing offloading utility function; The objective function is constructed based on the power peak load regulation utility function and the computational load shedding utility function.

3. The vehicle-network multi-dimensional resource elastic scheduling method based on hierarchical reinforcement learning according to claim 2 is characterized in that: The constructing of the power peak load regulation utility function comprises: Based on the reward obtained by the target CS for completing the peak-shaving task, the peak-shaving fee that the target CS needs to pay to the target ICV, and the penalty cost incurred by the target CS for failing to complete the peak-shaving task, the power peak-shaving utility function of the target CS is constructed; The constructing of the computation offloading utility function comprises: Based on the reward obtained by the target CS for completing the computing task, the computing fee that the target CS needs to pay to the target ICV, and the penalty cost incurred by the target CS for not completing the computing task, the power peak-shaving utility function of the target CS is constructed.

4. The vehicle-network multi-dimensional resource elastic scheduling method based on hierarchical reinforcement learning according to claim 2 is characterized in that: The calculation formula of the objective function is as follows: ； in, represents the objective function, Indicates the CS-ICV-user end matching relationship. Indicates the computing power value that ICV needs to provide. Indicates the charging or discharging power that the ICV needs to provide. Indicates Peak load scheduling cycle, Indicates the total number of peak-load scheduling cycles within the target scheduling period. Indicates The first peak load scheduling cycle Computation scheduling cycle, Indicates CS, Indicates the total number of CS in the target area, Indicates ICV, Indicates the total number of ICVs in the target area. Indicates A user terminal, Indicates the total number of users in the target area. Indicated in The peak load scheduling cycle CS and The matching relationship between ICVs, Indicated in The peak load scheduling cycle CS and The matching relationship of each user end, Indicated in In the peak load scheduling cycle The calculation scheduling cycle ICV and The matching relationship of each user end, Indicated in The peak load scheduling cycle CS calls The total revenue obtained by each ICV from completing peak load regulation and user-side computing tasks; Indicated in The peak load scheduling cycle CS calls The revenue obtained by ICVs from peak load shifting is Indicated in In the peak load scheduling cycle The calculation scheduling cycle CS calls ICV completed the The profit obtained from the computing task of each user.

5. The vehicle-network multi-dimensional resource elastic scheduling method based on hierarchical reinforcement learning according to claim 1 is characterized in that: The constraints include: in any peak-shaving scheduling cycle, the number of target ICVs participating in power peak-shaving shall not exceed the number of available target CSs; in any peak-shaving scheduling cycle, each target ICV can only select at most one target CS to participate in power peak-shaving; in any computing scheduling cycle in any peak-shaving scheduling cycle, the computing task of each user end can only be assigned to at most one target ICV; each target ICV must reach the corresponding target CS within a preset time threshold; each target ICV needs to have sufficient power to reach the corresponding target CS; in the peak-shaving stage, each target ICV needs to have sufficient power supply capacity after reaching the corresponding target CS; in the valley-filling stage, each target ICV needs to have sufficient charging gap after reaching the corresponding target CS; the charging or discharging power provided by each target ICV is within the preset power threshold; the computing power value provided by each target ICV is within the preset computing power threshold.

6. The vehicle-network multi-dimensional resource elastic scheduling method based on hierarchical reinforcement learning according to claim 1 is characterized in that: The parameters of the intelligent agent include observation state, action and reward; the observation state includes upper-level state and lower-level state, the upper-level state refers to the observation state of the target CS within the peak-shaving scheduling cycle, and the lower-level state refers to the observation state of the target CS within the calculation scheduling cycle in the peak-shaving scheduling cycle; the action includes upper-level action and lower-level action, the upper-level action refers to the power peak-shaving action taken by the target CS within the peak-shaving scheduling cycle, and the lower-level action refers to the calculation scheduling action taken by the target CS within the calculation scheduling cycle; the reward includes upper-level reward and lower-level reward, the lower-level reward refers to the calculation unloading benefit of the target CS within the peak-shaving scheduling cycle, and the upper-level reward refers to the power peak-shaving benefit and calculation unloading benefit of the target CS within the peak-shaving scheduling cycle.

7. The vehicle-network multi-dimensional resource elastic scheduling method based on hierarchical reinforcement learning according to claim 6 is characterized in that: The upper-level status includes: the peak-shaving capability of the target ICV, the computing capability of the target ICV, the driving distance from the target ICV to the target CS, the type of computing tasks that the target ICV is good at, peak-shaving instructions, the number of charging piles of the target CS, the type of the current peak-shaving phase, the price of the unit electricity involved in the peak-shaving, the matching relationship between the target user terminal and the target CS, and the basic information of the computing tasks of the target user terminal; the lower-level status includes: the peak-shaving capability of the target ICV, the computing capability of the target ICV, the upper-level action, the type of computing tasks that the target ICV is good at, the Poisson distribution parameters of the computing tasks, the matching relationship between the target user terminal and the target CS, and the basic information of the computing tasks of the target user terminal.

8. The vehicle-network multi-dimensional resource elastic scheduling method based on hierarchical reinforcement learning according to claim 6 is characterized in that: The upper-layer actions include the matching relationship between the target ICV and the target CS, and the charging or discharging power that the target ICV needs to provide to participate in peak regulation; the lower-layer actions include: the matching relationship between the target ICV and the target user terminal, and the computing power value that the target ICV needs to provide to participate in computing tasks.

9. The vehicle-network multi-dimensional resource elastic scheduling method based on hierarchical reinforcement learning according to claim 1 is characterized in that: The CS-ICV-user terminal interaction process includes: During the peak-shaving scheduling period, each target CS selects a target upper layer action according to the current upper layer state of each target CS, and the target ICV matching each target CS moves to each target CS for charging or discharging; In the computing scheduling period in the peak-shaving scheduling period, each target CS selects a target lower-layer action according to the current lower-layer state of each target CS, and the target ICV matching each target CS completes the computing task of the target user end.

10. The vehicle-network multi-dimensional resource elastic scheduling method based on hierarchical reinforcement learning according to claim 1 is characterized in that: The training process of the multi-agent model includes: Initializing the multi-agent model; Acquire an upper layer local observation state, and select an upper layer action according to the upper layer local observation state; Execute the above-mentioned upper-level actions and obtain upper-level rewards; Initialize the matching relationship between the lower-level global state and CS, ICV, and user end; According to the current global state of the lower layer, obtain the local observation state of the lower layer; Selecting a lower-layer action according to the current lower-layer global state and the matching relationship between the current CS, ICV, and user terminal; Execute the lower-level action, obtain the lower-level reward, the next lower-level global state, and the end signal; Store experience in the lower experience pool; Randomly extract data samples from the lower experience pool, train the lower Critic network and the lower Actor network, and update the lower target network every n steps, where n is a natural number greater than or equal to 1; Update upper-level rewards and store new experience in the upper-level experience pool; Data samples are randomly extracted from the upper experience pool, the upper Critic network and the upper Actor network are trained, and the upper target network is updated every n steps.

Images