A joint optimization method for vehicular network content caching and power allocation based on fairness.

Abstract

This invention relates to the field of edge caching technology in the Internet of Vehicles (IoV) and provides a fairness-based joint optimization method for IoV content caching and power allocation. The method includes: mapping content caching and content provider selection to discrete action sets, and power allocation to continuous actions, and associating discrete and continuous actions; mapping the relationship between the content caching stage and the content delivery stage to the association between discrete and continuous actions; introducing a fairness level α and establishing a system utility function based on the fairness level α; defining content value to quantify content popularity and priority, and defining the reward function of the P-DQN algorithm based on the content value and the system utility function; collecting environmental information and defining the state space of the P-DQN algorithm based on the environmental information; and training the action space, reward function, and state space using the P-DQN algorithm. This invention eliminates the need for relaxation of discrete actions or discretization of continuous actions.

Description

Technical Field

[0001] This invention relates to the field of edge caching technology for vehicle-to-everything (V2X) networks, and specifically to a joint optimization method for V2X content caching and power allocation based on fairness. Background Technology

[0002] Edge caching technology shows great potential in meeting the requirements of rapid response and low latency in connected vehicles (V2V). In V2V scenarios, vehicle-initiated content requests are often repetitive; that is, highly popular content (such as trending news, popular short videos, and real-time location information) is requested in large numbers by the same or different vehicles. This repeated transmission of the same content across the network wastes network resources, degrades overall system performance, and can even lead to network congestion. Edge caching addresses this by storing this highly popular content on edge nodes such as macro base stations and roadside units. When a vehicle user initiates a content request, it doesn't need to retrieve the content from the cloud server every time. With the content cached at the edge node, it can be delivered directly to the requesting vehicle, significantly reducing the traffic load on the backhaul link. By reducing the transmission of duplicate traffic in the core network, edge caching technology can provide low-latency content delivery services for vehicle users.

[0003] Because edge nodes have limited storage capacity and power allocation, and the traffic flow within their coverage area is extremely high, the power resources of these edge nodes cannot meet all content requests due to the large number of vehicles connecting to them. With the rapid development of deep reinforcement learning, deep reinforcement learning algorithms can be applied to content caching and power allocation methods. Traditional deep reinforcement learning-based optimization algorithms, when dealing with both discrete and continuous action spaces, often use discretization of continuous actions and relaxation of discrete actions. However, discretizing the continuous action space increases the complexity of the problem and reduces accuracy, while relaxing the discrete action space also increases complexity, making convergence difficult. Furthermore, optimization methods that only aim to minimize system latency or maximize throughput often neglect fairness considerations for the vehicular network system. This can lead to significant differences in network experience among vehicle users and prevent high-priority and popular content from being promptly responded to and delivered to requesting vehicles.

[0004] Meanwhile, when studying content caching and power allocation optimization strategies for vehicle-to-everything (V2X) systems, the following issues also need to be considered: First, the content in V2X systems changes over time, and when caching content, the dynamic changes in content popularity should be considered under the constraint of limited storage resources; second, there are massive numbers of vehicles and content in V2X systems, which not only brings the "curse of dimensionality" problem to optimization strategies, but also makes it easy to cause fairness issues when allocating limited resources.

[0005] In the context of vehicular network (V2V) edge caching, many simple and practical basic caching strategies already exist, such as Least Recently Used (LRU), Least Frequently Used (LRU), and First-In-First-Out (FIFO). These strategies are all derived from traditional caching scenarios. However, in the V2V scenario, the high-speed mobility of vehicles leads to rapidly changing network topologies, and the popularity of massive amounts of content is time-varying. Furthermore, existing V2V edge caching solutions rarely consider system fairness, resulting in poor network experience for some vehicle users and the inability to respond to high-value content in a timely manner.

[0006] Even when system fairness is considered, most existing research solutions for fairness in connected vehicle scenarios use logarithmic utility functions to achieve proportional fairness. However, using this logarithmic utility function can only provide a fixed compromise between system efficiency and fairness, which cannot adjust for preferences on the level of system efficiency and fairness. Summary of the Invention

[0007] In view of this, the purpose of the present invention is to provide a joint optimization method for vehicular network content caching and power allocation based on fairness.

[0008] This invention is achieved through the following technical solution:

[0009] A fairness-based joint optimization method for vehicular network content caching and power allocation includes:

[0010] S1. Based on the vehicle-to-everything (V2X) edge caching and content delivery scenarios, establish a system latency model for the V2X system. The optimization objective of the system latency model is to minimize system latency.

[0011] S2. Map the selection of content caching and content providers to discrete action sets, and power allocation to continuous actions, and associate discrete actions and continuous actions to obtain the initial action space;

[0012] S3. Map the relationship between the content caching stage and the content delivery stage to the relationship between discrete actions and continuous actions, and define the initial action space as the action space of the P-DQN algorithm according to the associated content.

[0013] S4. Introduce a fairness level α, establish a system utility function based on the fairness level α, and transform minimizing system latency into a trade-off control of the efficiency and fairness of the vehicle network system by maximizing the system utility function;

[0014] S5. Define content value to quantify content popularity and priority, and define the reward function of the P-DQN algorithm based on the content value and the system utility function;

[0015] S6. Collect environmental information and define the state space of the P-DQN algorithm based on the environmental information;

[0016] S7. The action space, reward function, and state space are trained using the P-DQN algorithm to obtain a joint optimization model for caching popular content and allocating content transmission power based on user fairness and content fairness. The optimal caching and power allocation strategy under a specified fairness level is obtained through the joint optimization model.

[0017] As an optimization, the specific steps of S1 are as follows:

[0018] S1.1 Establishing a vehicle-to-everything (V2X) system model: The V2X system includes a macro base station MBS, L... N Roadside Unit RSU and K N In this vehicle network system, the macro base station (MBS) can retrieve any content from the cloud data center (CDC) via a backhaul link. Simultaneously, the intelligent agent deployed on the MBS makes decisions based on the current state information and distributes these decisions to each roadside unit (RSU) communicatively connected to the MBS. Each RSU then performs corresponding actions based on the decision information. The vehicle network system operates on a time-slot basis, dividing the time axis into T segments. N There are time slots with a duration of Δ, where t is the time index, t∈{1,2,…,T}. N};

[0019] S1.2. Establish a content model for the aforementioned vehicle-to-everything (V2X) system model: The content model includes the probability p(n) of vehicle k requesting content f under the condition that vehicle k initiates a content request in time slot t. f (t)=1|n k (t) = 1), Popularity u of content f f (t) and the cache hit rate z of the roadside unit RSU; where, n f (t)∈{0,1},n k (t)∈{0,1},n f (t) = 1 indicates that content f is requested, n f (t) = 0 indicates that content f was not requested, n k nk(t) = 1 indicates that vehicle k initiated a content request, and nk(t) = 0 indicates that vehicle k did not initiate a content request. To characterize the skewness parameter of the Zipf distribution, V(f) represents the popularity ranking of content f, and F... N Number the total number of contents. This represents the weighted ranking value based on content popularity; cache hit rate. Among them, J lH represents the total number of content requests initiated by vehicles within the coverage area of ​​the l-th roadside unit (RSU). l (j) is the indicator function, H l (j) = 1 indicates that the l-th roadside unit (RSU) l When a vehicle within the coverage area requests content for the j-th time, the l-th roadside unit (RSU) l The requested content has been cached;

[0020] S1.3 Construct a transmission delay model for the aforementioned vehicle-to-everything (V2X) system model:

[0021]

[0022] Case 1 is: the l-th roadside unit (RSU) l The content f of vehicle k has been cached. For the l-th roadside unit (RSU) l The transmission delay s for directly sending content f to vehicle k f R is the size of the content f. l，k =Blog2(1+SINR) l，k ), which is the l-th roadside unit (RSU). l The maximum achievable transmission rate for sending content f to vehicle k; SINR l，k For the l-th roadside unit (RSU) l The signal-to-noise ratio between the signal and vehicle k;

[0023] Case 2 is: the l-th roadside unit (RSU) l The content f of vehicle k is not cached; the decision information requires the l-th roadside unit (RSU). l The content f is cached in this roadside unit, and the l-th roadside unit RSU l Neighboring roadside unit RSU 1′ Cached content f, For the neighbor's roadside unit RSU l′ Transmit content f to the l-th roadside unit (RSU) l The transmission delay, R l′，l For the neighbor's roadside unit RSU l′ The transmission rate between the l-th roadside unit RSUl and the l-th roadside unit RSUl The transmission delay for the l-th roadside unit (RSU) to directly send content f to vehicle k;

[0024] Case 3 is: the l-th roadside unit (RSU) l The content f of vehicle k is not cached; the decision information requires the l-th roadside unit (RSU). l The content f is cached in this roadside unit, but the neighboring roadside units RSU of the l-th roadside unit RSU l′Uncached content f, For the macro base station MBS to transmit content f to the l-th roadside unit RSU l The transmission delay, R MBS For macro base station MBS and the lth roadside unit RSU l The transmission rate between them The transmission delay for the l-th roadside unit (RSU) to directly send content f to vehicle k;

[0025] Case 4 is: the l-th roadside unit (RSU) l The content f of vehicle k is not cached, and the decision information does not require the l-th roadside unit (RSU). l Cache the content f in this roadside unit. For the macro base station MBS to transmit content f to vehicle k, R MBS，k The transmission rate between the macro base station MBS and vehicle k;

[0026] S1.4 Construct a queuing delay model for the aforementioned vehicle-to-everything (V2X) system model:

[0027]

[0028] Where x∈[1,2,3,4], x is 1, which corresponds to case A; x is 2, which corresponds to case B; x is 3, which corresponds to case C; x is 4, which corresponds to case D; and content f enters the data queue Q at time t0.

[0029] Case A is: the l-th roadside unit (RSU) l The content f of vehicle k has been cached and sent to vehicle k. For the l-th roadside unit (RSU) l Queuing delay for directly sending content f to vehicle k;

[0030] s i and R i Let q(t0) be the size of the i-th content in the data queue Q and the transmission rate of the transmitted content i, respectively; let q(t0) be the number of files in the data queue Q; let τ be the integral variable; and let δ be the length of a time slot during system operation, i.e., the time slot size.

[0031] Case B is: the l-th roadside unit (RSU) l The content f of vehicle k that is not cached is handled by the neighboring roadside unit RSU. l′ Send content f to the l-th roadside unit (RSU). l Then by the l-th roadside unit RSU l Send content f to vehicle k;

[0032] Case C is: the l-th roadside unit (RSU)l The content f of vehicle k that is not cached is sent by the macro base station MBS to the l-th roadside unit RSU. l Then by the l-th roadside unit RSU l Send content f to vehicle k;

[0033] The queuing delay for the macro base station MBS to send content f to the l-th roadside unit RSU1, The queuing delay for the l-th roadside unit RSUl to send content f to vehicle k;

[0034] Scenario D is: The macro base station MBS directly sends content f to vehicle k. Queuing delay for macro base station MBS to directly send content f to vehicle k.

[0035] As an optimization, the decision information includes deciding whether to cache the content in the RSU during the content caching phase and which specific content provider will send the content to the requesting vehicle at an appropriate transmit power during the content delivery phase.

[0036] As an optimization, the discrete action d in the discrete action set includes: whether the local RSU caches the requested content in the local RSU and which content provider sends the content to the requesting vehicle, the content provider including macro base station MBS and roadside unit RSU;

[0037] The continuous action p represents the transmission power.

[0038] The discrete and continuous actions constitute decision information, i.e., decision actions, which are represented as: a = (d, p); the initial action space is: A = {(d, p)} d )|d∈D,p d ∈P d}, where D∈{0,1,2} is a discrete action set, P d For a continuous set of actions of power magnitude sent by a content provider, (d, p) d ) indicates that the discrete action d corresponds to the continuous action p. d .

[0039] As an optimization, the specific process of S3 is as follows:

[0040] Define the association as follows:

[0041] (d=0,b d This indicates that the local roadside unit (RSU) where the requesting vehicle is located has cached the request content f, and the local RSU is transmitting it at a power of b. d Deliver content f to the requesting vehicle;

[0042] (d=1,b d This indicates that the local roadside unit (RSU) has not cached content f. The local RSU needs to first obtain and cache content f from its neighboring RSU or MBS, and then transmit it at power b. d The content f is delivered to the requesting vehicle. If the neighboring roadside unit (RSU) has content f cached, the local roadside unit (RSU) obtains content f from the neighboring roadside unit (RSU); otherwise, it obtains content f from the macro base station (MBS).

[0043] (d=2,b d If the local roadside unit (RSU) has not cached the requested content f, and content f is unlikely to become popular content within the coverage area of ​​the local RSU, then the macro base station (MBS) will transmit the request at power b. d Deliver content f to the requesting vehicle.

[0044] As an optimization, the system utility function is:

[0045]

[0046] in, Represents the fair utility function, τ represents the throughput of vehicle k. k，f (t) represents the time taken from when vehicle k initiates a request for content f until the vehicle finally accepts and completes the request; n k，f (t)∈{0,1}, representing the state of vehicle k initiating a request for content f in time slot t, where n k，f (t) = 1 indicates that vehicle k initiated a request for content f, n k，f (t) = 0 indicates that vehicle k did not initiate a request for content f, c l，f This indicates that content f is in RSU l Cache status in p k This represents the power allocated when transmitting the content requested by vehicle k, where T is the number of time slots.

[0047] As an optimization, in S4, the system utility function is maximized to transform minimizing system latency into a trade-off control between the efficiency and fairness of the vehicle-to-everything (V2X) system. The specific optimization problem is set as follows:

[0048] As an optimization, in section 5, the content value of content f is specifically: η f =iu f +(1-ι)o f / O N Where ι∈[0,1] represents the popularity weighting factor, o f∈{0, 1, 2, ..., O N} represents the priority of content f, O N The priority level number, where o f =O N This indicates that content f has the highest priority.

[0049] As an optimization, in S6, the environmental information includes content cache status, content request status, requested content size, content value, and channel gain. The state space of the P-DQN algorithm is:

[0050] S = {C, M, H, α}, where... Indicates the content cache status; c l，f =1 indicates the first roadside unit (RSU) l The cached content f and c l，f =0 indicates the first roadside unit (RSU) l For cached content f;

[0051] M = {f, s} f η f} represents the information of the requested content, and f and s f These represent the index of the requested content and the size of the content, respectively. The index of the requested content is the content number, η. f ∈[0,1] represents the content value of content f;

[0052] H = {h} l，k h MBS，k} represents the channel gain, h l，k For the first roadside unit (RSU) l The channel gain h between vehicle k and vehicle k MBS，k The channel gain between the macro base station MBS and vehicle k;

[0053] α represents the fairness level of the vehicle-to-everything (V2X) system.

[0054] As an optimization, the specific steps for S7 are as follows:

[0055] S7.1 Using a deterministic policy network p d (s t ;θ t Generate a series of continuous action values ​​p based on each state in the state space S. d , where θ t For deterministic policy network p d (s t ;θ t Network weights, s t These are the states in the state space;

[0056] S7.2, the obtained continuous action value pd Together with the current state value s t Input to deep Q network Train and select the discrete action d with the largest Q value. t ,in For deep Q network Network weights;

[0057] S7.3 Use an ∈-greedy strategy to increase the probability of action exploration;

[0058] S7.4, When performing mixed actions Then calculate latency and cache hit rate, and update the state to s. t+1 Simultaneously calculate the instant reward r t ;

[0059] S7.5, The quadruple (s t a t r t s t+1 The mini-batch set, a, is stored in the experience replay pool Γ and sampled from the experience replay pool. t For time slot t, the decision action;

[0060] S7.6, Update network parameters And θ, finally we get the joint optimization model;

[0061] S7.7 Given the fairness level in the state space, the optimal cache and power allocation strategy under the specified fairness level is finally obtained.

[0062] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0063] The joint optimization algorithm for content caching and power allocation in the Internet of Vehicles proposed in this invention incorporates considerations of system fairness into the optimization scheme that only aims to minimize system latency, thereby avoiding poor network experience for some users and the failure to respond to high-value content in a timely manner.

[0064] Using content value to quantify the relationship between content popularity and priority, and leveraging the maximization of the system utility function to achieve fairness, the original optimization goal of minimizing system latency is transformed into a trade-off control of system efficiency and fairness.

[0065] The selection of content caching and content providers is mapped to a discrete set of actions, while power allocation is used as a continuous parameter to associate with discrete actions. This problem with a mixed discrete-continuous action space is solved by parametric deep reinforcement learning algorithms, thus eliminating the need to relax discrete actions or discretize continuous actions. Attached Figure Description

[0066] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings:

[0067] Figure 1 This is a flowchart of a fairness-based joint optimization method for content caching and power allocation in vehicle networking according to the present invention;

[0068] Figure 2 A model diagram for caching and delivering connected vehicle content;

[0069] Figure 3 This is a flowchart of the P-DQN algorithm. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0071] An embodiment of a fairness-based joint optimization method for vehicle-to-everything (V2X) content caching and power allocation, such as... Figure 1 As shown, it includes the following steps:

[0072] S1. Based on the vehicle-to-everything (V2X) edge caching and content delivery scenarios, establish a system latency model for the V2X system. The optimization objective of the system latency model is to minimize system latency.

[0073] Furthermore, step S1 specifically includes the following steps:

[0074] Step 1.1: Construct the system model. For example... Figure 2 As shown, the system model in this invention mainly consists of macro base stations (MBS) and roadside units (RSUs) L = {1, 2, ..., L}. N} and vehicles K = {1,2,...,K N The MBS (Multi-Level Service) consists of several components. MBS can retrieve any popular content from the system's content library via a backhaul link from the Cloud Data Center (CDC). Local RSU (Remote Subscriber Unit) is relative to the requesting vehicle; if vehicle k is in the RSU... l If a content request is initiated within the signal coverage area, then RSU lThis refers to the local RSU requesting vehicle k. The local RSU communicates with its neighboring RSUs via wired connections such as fiber optics, and both can obtain popular content from the MBS (macro base station) or its neighboring RSUs and cache it locally. Wireless communication connections based on the C-V2X standard can be established between RSUs and MBS, between RSUs and vehicles, and between MBS and vehicles. RSUs collect state information and send it to MBS. This state information includes content request status, content caching status, and channel status. Agents deployed on MBS make decisions based on the state information and distribute the decision information to each RSU, which then execute corresponding actions. The decision information includes content caching and power allocation: on the one hand, if the decision information requires the local RSU to cache the requested content locally, then if the neighboring RSU has already cached the requested content, the local RSU obtains the content from the neighboring RSU and caches it locally; if the neighboring RSU has not cached the content, the local RSU obtains the content from the MBS and caches it locally. This process is the content caching phase. On the other hand, the decision information requires the RSU or MBS to deliver the requested content to the requesting vehicle at an appropriate transmit power; this process is the content delivery phase. The system operates on a time-slot basis, dividing the time axis into T... N There are time slots with a duration of Δ, t∈{1,2,...,T}. N} represents the time slot index.

[0075] Step 1.2: Construct the content model. Define a content library set F = {1, 2, ..., F}. N The size of the content f∈F is defined as s. f If vehicle k issues at most one content request within a time slot with probability p∈(0,1), then the probability of k issuing a content request in time slot t is Pn. k t = 1 = p, where n k t=1 indicates that a vehicle initiated a content request. Given that vehicle k initiated a content request in time slot t, the probability that vehicle will request content f is pn. f t = 1 | n k t = 1 = u f t, where n f (t)∈{0,1},n f (t) = 1 indicates that content f is requested. The probability of initiating a request involves the vehicle's content request model (i.e., the vehicle initiates a content request with a certain probability). Different request models will have different impacts on the prediction results of the system model. The popularity of content f is defined. Where k∈[0.6, 1.2] is the skewness parameter characterizing the Zipf distribution, and V(f) is the popularity ranking of content f; This represents the weighted ranking value of content popularity, and the content popularity of content f is defined as π. fThe content f is π for each request. f Add 1; aggregate content popularity Arranging the content in descending order yields sequence H. The content with the highest popularity is placed in the first position of sequence Π. The index of content f in sequence H is defined as the popularity ranking V(f) of that content.

[0076] Define the cache hit rate z as: J l RSU l The total number of content requests initiated by vehicles within the coverage area, H l (j) is the indicator function, H l (j) = 1 indicates that RSU l For vehicles within the coverage area, during the j-th content request, RSU l The requested content has been cached.

[0077] The cache hit rate has two functions in this invention. The first function is that the cache hit rate needs to be used in the reward function during model training. The second function is to evaluate the performance.

[0078] Whether to cache is determined by directly reading the cache status of each item in the RSU, using the symbol 'c'. l，f It means, c l，f =1 indicates RSU l There are cached contents f and c l，f =0 indicates RSU l No cached content f.

[0079] Step 1.3: Construct the transmission delay model. Define the total downlink transmission delay of sending f from its location (MBS or RSU) to vehicle k at time slot t as: but There are four possible scenarios:

[0080] (1) Case 1: If RSU l If the cached content f is used, then RSU l With the maximum achievable transmission rate R l，k =Blog2(1+SINR) l，k Send content f to vehicle k, where SINR l，k For RSU l The signal-to-interference-to-noise ratio between k and . In this case, Equal to RSU l Transmission delay of sending f to vehicle k Right now

[0081] (2) Case 2: If the decision action requires RSU lCache content f locally, and RSU l neighboring RSU l′ f has been cached. Due to RSU l with RSU l′ They communicate via optical fiber. To simplify the analysis, the transmission rate between them is set to R. l′，l It is a fixed value. Therefore, RSU l First from RSU l′ f is obtained at point f, and its transmission delay is Then RSU l Send f to the requesting vehicle k; the transmission delay is... Then there is

[0082] (3) Scenario 3: If the decision action requires RSU l The content f is cached locally, but its neighboring RSUs l′ If f is not cached, then calculate RSU. l First, f is obtained from MBS, and its transmission delay is... Where R MBS，l MBS will send f to RSU l The transmission rate, then RSU l Send f to the requesting vehicle k, then

[0083] (4) Case 4: If the decision action does not require RSU l If content f is cached locally, MBS will send f to the requesting vehicle f. And τ MBS，k =s f / R MBS，k For MBS at a transmission rate R MBS，k The transmission delay of sending f to vehicle f.

[0084] Step 1.4: Queuing Delay Model. Define the MBS sending content to the RSU. l The virtual data transmission queues for vehicles are respectively and RSU l The virtual data transmission queue for sending content to the vehicle is Q. l If content enters the data queue Q at time t0, the expected queuing delay for that content can be calculated. Where q(t0) is the number of files in the data queue Q, and s i and R i Let be the size of the i-th content in the queue and the transmission rate of content i, respectively. Let t0 be the start time of time slot t, then the expected average queuing delay of queue Q in time slot t is... τ is the integration variable, and δ is the length of one time slot during system operation, i.e., the size of the time slot, corresponding to the four cases of transmission delay in step 1.3:

[0085] In scenario one, RSU l If content f is sent to vehicle k at time slot t, then there will be a queuing delay in this case. In scenario two, the neighboring RSU sends f to the local RSU, and then the local RSU sends f to vehicle k. Since the queuing delay from the neighboring RSU to the local RSU is ignored, the queuing delay in scenario two is: In scenario three, MBS sends f to the local RSU, with a queuing delay of [missing information]. The local RSU then sends f to the vehicle, with a queuing delay of [time period missing]. The queuing delay in case three is: TQlMBSt. In case four, the MBS sends the content to the vehicle, and the queuing delay is...

[0086] The formulas for these three parameters are all They actually represent three queues of data. Q1 represents MBS to RSU respectively. l The queuing system, the MBS to vehicle queuing system, and the RSU l The queue of vehicles.

[0087] S2. Map the selection of content caching and content providers to discrete action sets, and power allocation to continuous actions, and associate discrete actions and continuous actions to obtain the initial action space;

[0088] Furthermore, step S2 specifically includes the following steps:

[0089] In the system model of this invention, the agent deployed on the MBS makes decisions based on vehicle request events. Specifically, in response to a content request event initiated by a vehicle, the agent makes a decision based on the state space. This decision instructs the local RSU whether to cache the requested content locally, indicates which content provider (RSU or MBS) should deliver the content to the requesting vehicle, and specifies the transmission power for delivering the requested content. A discrete action `d` determines whether the local RSU caches the requested content locally and whether the MBS or the local RSU sends the content to the requesting vehicle, while a continuous action `p` determines the transmission power. Therefore, the decision action can be defined as a = (d, p). Furthermore, the discrete and continuous actions are associated, with the continuous parameter corresponding to the discrete action `d` being `p`. d Then define the action space A = {(d, p} d)|d∈D,p d ∈P d}, where D = {0, 1, 2} is the discrete action set.

[0090] S3. Map the relationship between the content caching stage and the content delivery stage to the relationship between discrete actions and continuous actions, and define the initial action space as the action space of the P-DQN algorithm according to the associated content.

[0091] Furthermore, step S3 specifically includes the following steps:

[0092] (d=0,p d This indicates that the local RSU where the requesting vehicle is located has cached the request content f, and the local RSU is transmitting it at a power of p. d Deliver f to the requesting vehicle. (d = 1, p) d This indicates that the local RSU has not cached the content f. The local RSU needs to obtain and cache f from the neighboring RSU or MBS first, and then transmit the f at the specified power p. d Deliver f to the requesting vehicle. If the neighboring RSU has f in its cache, the local RSU retrieves f from the neighboring RSU; otherwise, it retrieves f from the MBS. (d = 2, p) d If the local RSU does not cache the requested content f, and f is not likely to become popular content within the RSU coverage area, then the MBS will transmit it at power p. d Deliver f to the requesting vehicle. Determining whether content qualifies to become popular content is what the system model does; essentially, it optimizes the variables in the objective. In other words, if the system (or model) believes that a certain piece of content should become popular content, then that content is cached in RSU.

[0093] S4. Introduce a fairness level α, establish a system utility function based on the fairness level α, and transform minimizing system latency into a trade-off control of the efficiency and fairness of the vehicle network system by maximizing the system utility function;

[0094] Furthermore, step S4 specifically includes the following steps:

[0095] If we construct the fairness problem under consideration using a fairness utility function based on the fairness level α, then the α fairness utility function is defined as follows:

[0096]

[0097] Where x is a performance parameter. The throughput of vehicle k is introduced. Where s f Let τ be the size of the content f. k，f(t) represents the time taken from when vehicle k initiates a request for file f until the vehicle finally receives and completes the request. The α utility function can be expressed as:

[0098]

[0099] Here, α can be used to adjust the system's preference for throughput or user fairness; a larger α indicates a higher level of service fairness among users. To balance optimizing the overall average throughput of the vehicle-to-everything (V2X) system with considering the fairness of system resource allocation, a system utility function is defined. Where n k，f (t)∈{0,1} represents the state of vehicle k initiating a request for content f in time slot t, where n k，f (t) = 1 indicates that vehicle k initiated a request for content f; conversely, n k，f If (t) = 0, it means that vehicle k did not initiate a request for content f. l，f This indicates that content f is in RSU l Cache status in p k Let represent the power allocated when transmitting the requested content from vehicle k. Therefore, the system utility function maximization problem based on α-fairness is modeled as an optimization problem.

[0100] S5. Define content value to quantify content popularity and priority, and define the reward function of the P-DQN algorithm based on the content value and the system utility function;

[0101] Furthermore, step S5 specifically includes the following steps:

[0102] Define the priority of content f as o f ∈{0,1,...,O N}, where o f =O N This indicates that content f has the highest priority. The content value η of content f is defined. f =ιu f +(1-ι)o f / O N , where ι∈[0,1] represents the popularity weight factor, which represents the system's preference for content popularity and content priority. When ι=1, it means that only content popularity is considered as content value.

[0103] Content priority and content popularity are different. Content popularity is objective; for example, if certain content is accessed very frequently, we can say that its popularity is high, which is quantifiable. Content priority, on the other hand, is defined based on the social attributes or transmission requirements of the content. For example, real-time maps, traffic information, or important warning messages may not necessarily have high popularity (i.e., low access frequency) at all times, but compared to highly popular content (such as short videos or other entertainment information), this type of information is certainly more important from a practical perspective. In other words, content priority is pre-set by humans, based on their perceived importance.

[0104] S6. Collect environmental information and define the state space of the P-DQN algorithm based on the environmental information;

[0105] Furthermore, step S6 specifically includes the following steps:

[0106] When the content provider (RSU or MBS) delivers content to the requesting vehicle, the agent needs to be able to allocate reasonable power to the requesting vehicle. Therefore, the size of the requested content and the channel gain should also be included as components of the state space. To enable the agent to perceive the current system's preference between system efficiency and fairness during decision-making, the fairness level α should also be included as part of the state space. Therefore, the state space is defined as S = {C, M, H, α}, where... Indicates the content caching status. M = {f, s} f ,η f} represents the information of the requested content, and f and s f η represents the index of the requested content and the size of that content, respectively. f ∈[0,1] represents the content value of content f. H={h l,k ,h MBS,k} represents the channel gain, and h l,k For RSU l The channel gain h between vehicle k and vehicle k MBS,k This represents the channel gain between MBS and vehicle k.

[0107] S7. The action space, reward function, and state space are trained using the P-DQN algorithm to obtain a joint optimization model for caching popular content and allocating content transmission power based on user fairness and content fairness. The optimal caching and power allocation strategy under a specified fairness level is obtained through the joint optimization model.

[0108] Furthermore, step S7 specifically includes the following steps:

[0109] like Figure 3As shown, a deterministic policy network p(θ) is first used to generate continuous action values ​​based on the state S, where θ is the network weight of p(θ). Then, the continuous action values ​​along with the state S are input into a deep Q-network. Among them for The network weights are determined, and finally, the discrete action with the largest Q value and its corresponding continuous action are selected.

[0110] In conjunction with this invention, firstly, state s t Input to network p d (s t ;θ t In ) a series of continuous actions p are generated. d Then p d Along with state s t Input to network And select the discrete action d with the largest Q value. t To avoid the model getting trapped in local optima, after obtaining the optimal d... t Then, an ∈-greedy strategy is used to increase the probability of action exploration. This is applied when performing mixed actions. Then calculate latency and cache hit rate, and update the state to s. t+1 Simultaneously calculate the instant reward r t Next, the quadruple (s) t ,a t ,r t ,s t+1 The network parameters are stored in the experience replay pool Γ and sampled from the experience replay pool. Finally, the network parameters are updated. and θ.

[0111] The generated continuous action values ​​are not a single value, but a series, such as input states S to p. d (s t ;θ t In the network, the output continuous action values ​​are 0.621, 0.712, and 0.495. It should be noted that the three continuous action values ​​above correspond to discrete actions 0, 1, and 2, respectively.

[0112] Next, input the state S and a series of consecutive actions (0.621, 0.712, 0.495) together into the input field. In the network, the network will output 3 Q values. For example, the 3 Q values ​​output are 12, 17 and 11. The largest Q value here is 17, which corresponds to discrete action 1. The continuous action corresponding to the discrete action is 0.712. So the discrete action and the continuous action are finally selected as 1 and 0.712 respectively.

[0113] However, to reduce the probability of the model getting stuck in local optima, it is necessary to use "exploration probability" to select actions. Here, exploration probability refers to selecting a discrete action with a relatively small probability (e.g., 0.01). The previously mentioned method of selecting the discrete action with the largest Q-value, along with its corresponding continuous action, could lead to the model getting stuck in local optima. Therefore, when selecting discrete actions, instead of always choosing the one with the largest Q-value, we randomly select an action with a certain probability (e.g., 0.01). That is, the probability of selecting the action with the largest Q-value is 0.99, and the probability of randomly selecting a discrete action is 0.01.

[0114] Latency is used to calculate the throughput of vehicle k used later, that is, the time τ taken for vehicle k to initiate a request for content f and finally receive and complete the requested content. k，f (t), while throughput is used in the fair utility function;

[0115] Cache hit rate is used in the reward function to improve the system hit rate, and it can also be used in experimental evaluation.

[0116] In deep reinforcement learning, mini-batch refers to grouping multiple samples (usually a consecutive series of samples) into smaller groups, allowing for batch training within each group. This approach allows for more computation and gradient updates within each group, thus improving the algorithm's training speed and efficiency.

[0117] Mini-batch is a common data processing method and is not a concept unique to deep reinforcement learning. Many other machine learning fields, such as supervised learning, semi-supervised learning, and deep learning, use mini-batch for data processing and model training.

[0118] Network parameters are updated. In deep reinforcement learning, network weights are updated using the backpropagation algorithm. Backpropagation is a commonly used machine learning optimization algorithm used to calculate the gradient of the loss function with respect to the weights and update the weights based on this gradient. The specific backpropagation algorithm is integrated into the PyTouch toolkit. The backpropagation process is as follows:

[0119] Specifically, during training, the network output (i.e., the agent's behavior) is first calculated based on observed data (i.e., the current environmental state). Then, the expected reward signal corresponding to the output is calculated. Finally, the gradient of the loss function with respect to the network weight parameters is calculated. Next, optimization algorithms such as gradient descent are used to update the network weight parameters, reducing the value of the loss function. This process is typically repeated multiple times until the loss function converges to a stable value, completing one training cycle and updating the network parameters.

[0120] Specifically, the action-value function is expressed as Q(s,a)=Q(s,d,p) d ), where s∈S. The agent selects a discrete action at time t, p dt For discrete action d t The corresponding continuous parameter values ​​allow the Bellman equation to be expressed as:

[0121]

[0122] Where γ is the discount factor, r t This represents the immediate reward the agent receives at time t.

[0123] Using deep neural networks Approximating Q(s,d,p) d And utilizes a deterministic policy network p d (s;θ:S→P d Approaching With fixed network weights When, we want to find θ such that the equation This is valid. Combining the n-step algorithm, for a fixed n≥1, the n-step objective value y... t Defined as:

[0124]

[0125] It should be noted that "sup" here means supremacy, which is the least upper bound of a set (sequence or function).

[0126] In deterministic policy networks, the goal is to estimate the expected reward of an agent taking a certain action in a given state by optimizing the Q-function. There are several reasons for approximating the Q-value to its upper bound:

[0127] 1. Maximizing cumulative reward: In deterministic policy networks, the goal is to maximize the agent's cumulative reward throughout the training process. By approximating the Q-value to the upper bound of Q, the difference between the Q-value and the upper bound of Q can be minimized, thereby maximizing the cumulative reward.

[0128] 2. Reduce computational complexity: Approximating the Q value to the supremum of Q can significantly reduce computational complexity. This is because when updating policy parameters, it is necessary to calculate the difference between the Q value and the supremum of Q. When the Q value is close to the supremum of Q, this difference will become very small, thereby reducing computational complexity.

[0129] 3. Stable convergence: Approximating the Q value to the supremum of Q can ensure that the algorithm converges stably during training. This is because when the Q value is close to the supremum of Q, the difference between them is small, and it is easier to find a stable update step size, thereby causing the value of the loss function to converge to a smaller value.

[0130] It is important to note that approximating the upper bound of Q to the Q-value is not mandatory in practical applications. In fact, the Q-value is usually trained to be close to the mean of Q. However, in some special cases, such as zero-sum games or games with two possible actions, approximating the upper bound of Q to the Q-value may yield better results. The specific method used depends on the nature of the problem and the optimization objective.

[0131] right Use the least squares loss function. Furthermore, in order to... Find the fixed position Maximize θ, set The loss function for θ is:

[0132]

[0133]

[0134] The loss function types are as follows:

[0135] 1. Experience replay loss: This method accumulates experience from different state-action pairs and updates samples based on a specific strategy. The loss function typically calculates the mean or median of the accumulated experience.

[0136] 2. Online experience replay loss: Similar to experience replay, but the update step size is adapted to the rate of experience accumulation. The loss function typically calculates the instantaneous mean or median of accumulated experience.

[0137] 3. Q-target loss: In methods such as DQN, the Q-value prediction is compared with the Q-value target, and the difference between the two is calculated. The loss function typically calculates the mean of the difference.

[0138] 4. Policy gradient loss: Calculates the expected value of the policy gradient based on the probability distribution of actions in a given state. The loss function is usually the negative of the expected gradient.

[0139] 5. Discounted reward expectation loss: Calculates the expected future reward based on the discount effect of the agent's actions. The loss function typically calculates a negative value for the expected discounted reward.

[0140] When calculating the loss function, it is usually necessary to set an update step size or learning rate. The optimization algorithm updates the agent's policy parameters or value function parameters based on the loss function and the learning rate.

[0141] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A fairness-based joint optimization method for content caching and power allocation in vehicle-to-everything (V2X) networks, characterized in that, include: S1. Based on the vehicle-to-everything (V2X) edge caching and content delivery scenarios, establish a system latency model for the V2X system. The optimization objective of the system latency model is to minimize system latency. S2. Map the selection of content caching and content providers to discrete action sets, and power allocation to continuous actions, and associate discrete actions and continuous actions to obtain the initial action space; S3. Map the relationship between the content caching stage and the content delivery stage to the relationship between discrete actions and continuous actions, and define the initial action space as the action space of the P-DQN algorithm based on the associated content. The P-DQN algorithm is a parameterized deep reinforcement learning algorithm. S4. Introducing Fairness Levels Establish a system based on fairness levels The system utility function is determined, and the minimization of system latency is transformed into a trade-off control over the efficiency and fairness of the vehicle-to-everything (V2X) system by maximizing the system utility function. The system utility function is: ； in, Represents the fair utility function, , , representing the throughput of vehicle k. This represents the time taken from when vehicle k initiates a request for content f until the vehicle finally accepts and completes the request. The size of the content f; This indicates the state of vehicle k requesting content f in time slot t. This indicates that vehicle k initiated a request for content f. This indicates that vehicle k did not initiate a request for content f. Indicates content f in Cache status in the middle, This represents the power allocated when transmitting the content requested by vehicle k. Number the total number of contents. The total number of vehicles is numbered, and T is the number of time gaps; Maximizing the system utility function transforms minimizing system latency into a trade-off between efficiency and fairness in the vehicle-to-everything (V2X) system. The specific optimization problem is as follows: ; S5. Define content value to quantify content popularity and priority, and define the reward function of the P-DQN algorithm based on the content value and the system utility function; S6. Collect environmental information and define the state space of the P-DQN algorithm based on the environmental information; S7. The action space, reward function, and state space are trained using the P-DQN algorithm to obtain a joint optimization model for caching popular content and allocating content transmission power based on user fairness and content fairness. The optimal caching and power allocation strategy under a specified fairness level is obtained through the joint optimization model.

2. The method for joint optimization of vehicle network content caching and power allocation based on fairness according to claim 1, characterized in that, The specific steps of S1 are as follows: S1.1 Establishing a vehicle-to-everything (V2X) system model: The V2X system includes a macro base station (MBS). Roadside Units (RSUs) and In this vehicle network system, the macro base station (MBS) can retrieve any content from the cloud data center (CDC) via a backhaul link. Simultaneously, the intelligent agent deployed on the MBS makes decisions based on the current state information and distributes these decisions to each roadside unit (RSU) communicatively connected to the MBS. Each RSU executes corresponding actions based on the decision information. The vehicle network system operates on a time-slot basis, dividing the time axis into segments. The duration is The time slot, where t is the time index. ; S1.2 Establish a content model for the vehicle-to-everything (V2X) system model: The content model includes the probability that vehicle k requests content f given that vehicle k initiates a content request in time slot t. Popularity of content f And the cache hit rate z of the Roadside Unit (RSU); where, , , This indicates that content f was requested. This indicates that content f was not requested. This indicates that vehicle k initiated a content request. This indicates that vehicle k did not initiate a content request. , , is a skewness parameter characterizing the Zipf distribution. Rank content f based on its popularity. Number the total number of contents. This represents the total weighted ranking value based on the popularity of all content; cache hit rate. ,in, Indicates the first l The total number of content requests initiated by vehicles within the coverage area of ​​each Roadside Unit (RSU). For indicator functions, Indicates the first l roadside unit Vehicles within the coverage area, during the j-th request for content, l roadside unit The requested content has been cached; S1.3 Construct a transmission delay model for the aforementioned vehicle-to-everything (V2X) system model: ； Case 1 is as follows: l roadside unit The content f of vehicle k has been cached. For the first l roadside unit The transmission delay of directly sending content f to vehicle k. The size of the content f, , for the first l roadside unit The maximum achievable transmission rate for sending content f to vehicle k; For the first l roadside unit The signal-to-noise ratio between the signal and vehicle k; Scenario 2 is: l roadside unit The content f of vehicle k is not cached; the decision information requires the first... l roadside unit Cache content f in this roadside unit, and the first... l roadside unit Neighbor's roadside unit Cached content f, For the neighbor's roadside unit Transfer content f to the first l roadside unit Transmission delay, For the neighbor's roadside unit With the l roadside unit The transmission rate between them For the first l The transmission delay of a roadside unit (RSU) directly sending content f to vehicle k; Scenario 3 is: l roadside unit The content f of vehicle k is not cached; the decision information requires the first... l roadside unit Content f is cached in this roadside unit, but the first... l The neighboring roadside units of each roadside unit RSU Uncached content f, For the macro base station MBS to transmit content f to the first l roadside unit Transmission delay, For macro base station MBS and the first l roadside unit The transmission rate between them For the first l The transmission delay of a roadside unit (RSU) directly sending content f to vehicle k; Case four is: l roadside unit The content f of vehicle k is not cached, and the decision information does not require the first... l roadside unit Cache the content f in this roadside unit. For the macro base station MBS to transmit content f to vehicle k, The transmission rate between the macro base station MBS and vehicle k; S1.4 Construct a queuing delay model for the aforementioned vehicle-to-everything (V2X) system model: ； in, When x is 1, it corresponds to case A; when x is 2, it corresponds to case B; when x is 3, it corresponds to case C; when x is 4, it corresponds to case D. The content f is at time... Enter into data queue Q; Case A is: l roadside unit The content f of vehicle k has been cached and sent to vehicle k. For the first l roadside unit Queuing delay for directly sending content f to vehicle k; ; ; and Let be the size of the i-th element in data queue Q and the transmission rate of the transmitted element i, respectively. Let Q be the number of files in the data queue. For integration variables, The length of a time slot during system operation, i.e., the time slot size; Case B is: l roadside unit The content f of vehicle k that is not cached is obtained from the neighboring roadside unit. Send content f to the first l roadside unit Then by the first l roadside unit Send content f to vehicle k; Case C is: l roadside unit The content f of vehicle k that is not cached will be sent by the macro base station MBS to the first... l roadside unit Then by the first l roadside unit Send content f to vehicle k; For the macro base station MBS, content f will be sent to the first... l roadside unit Queuing delay, For the first l roadside unit The queuing delay for sending content f to vehicle k; Scenario D is: The macro base station MBS directly sends content f to vehicle k. Queuing delay for macro base station MBS to directly send content f to vehicle k.

3. The method for joint optimization of vehicle network content caching and power allocation based on fairness according to claim 2, characterized in that, The decision information includes deciding whether to cache the content in the RSU during the content caching phase and which content provider will send the content to the requesting vehicle at an appropriate transmit power during the content delivery phase.

4. The method for joint optimization of vehicle network content caching and power allocation based on fairness according to claim 3, characterized in that, The discrete action d in the discrete action set includes: whether the local RSU caches the requested content in the local RSU and which content provider sends the content to the requesting vehicle. The content providers include macro base stations (MBS) and roadside units (RSUs). The continuous action p represents the transmission power. The discrete and continuous actions constitute decision information, i.e., decision actions, which are represented as follows: ; The initial action space is: ,in, , is a discrete set of actions. A continuous set of actions representing the power of content sent by the content provider. This indicates that discrete action d corresponds to continuous action. .

5. The method for joint optimization of vehicle network content caching and power allocation based on fairness, as described in claim 4, is characterized in that... The specific process of S3 is as follows: Define the association as follows: This indicates that the local roadside unit (RSU) where the requesting vehicle is located has cached the request content f, and the local RSU is transmitting it at high power. Deliver content f to the requesting vehicle; This indicates that the local roadside unit (RSU) has not cached content f. The local RSU needs to first obtain and cache content f from its neighboring RSU or MBS, and then transmit the content f at the specified power. The content f is delivered to the requesting vehicle. If the neighboring roadside unit (RSU) has content f cached, the local roadside unit (RSU) obtains content f from the neighboring roadside unit (RSU); otherwise, it obtains content f from the macro base station (MBS). If the local roadside unit (RSU) has not cached the requested content f, and content f is unlikely to become popular content within the coverage area of ​​the local RSU, then the macro base station (MBS) will transmit the content at its maximum power. Deliver content f to the requesting vehicle.

6. The method for joint optimization of vehicle network content caching and power allocation based on fairness according to claim 1, characterized in that, In S5, the specific content value of content f is as follows: ,in Weighting factors representing popularity , where f is the priority of content. The priority level number, where This indicates that content f has the highest priority.

7. The method for joint optimization of vehicle network content caching and power allocation based on fairness according to claim 6, characterized in that, In S6, the environmental information includes content cache status, content request status, requested content size, content value, and channel gain. The state space of the P-DQN algorithm is: ,in, Indicates the content cache status; Represents the l-th roadside unit The content f has already been cached. Represents the l-th roadside unit The content to be cached is f; Information representing the requested content, and and These represent the index of the requested content and the size of the content, respectively. The index of the requested content is the content number. For content f The value of its content; Indicates channel gain. For the l-th roadside unit and vehicles k Channel gain between For macro base stations MBS and vehicles k Channel gain between; To determine the fairness level of the vehicle networking system.

8. The method for joint optimization of vehicle network content caching and power allocation based on fairness according to claim 7, characterized in that, The specific steps for S7 are as follows: S7.1 Using deterministic policy networks Generate a series of continuous action values ​​based on each state in the state space S. ,in For deterministic policy networks Network weights These are the states in the state space; S7.2, Obtain the continuous action values Together with the current state value Input to deep Q network Train and select the discrete action with the largest Q value ,in For deep Q network Network weights; S7.3 Use an ε-greedy strategy to increase the probability of action exploration; S7.4, When performing mixed actions Then calculate latency and cache hit rate, and update the state accordingly. Simultaneously calculate instant rewards , Discrete Actions Corresponding continuous parameter values; S7.5, quaternion Stored in the experience replay pool In, and the mini-batch set sampled from the experience replay pool, For time slot t, the decision action; S7.6, Update network parameters and Finally, a joint optimization model is obtained; S7.7 Given the fairness level in the state space, the optimal cache and power allocation strategy under the specified fairness level is finally obtained.

Images