Edge node selection scheme considering content placement

Abstract

The invention discloses an edge node selection scheme considering content placement, belongs to the technical field of mobile communication technologies and computer networks, and is used for solving the problem of effectively utilizing edge cache to shorten a transmission path on the premise of ensuring stable transmission in an edge cache environment of the Internet of Vehicles. Firstly, a network control center collects vehicle and network data at an edge node side formed by a base station and an edge server, and secondly, based on the collected data, a caching strategy matrix is solved, a caching scheme is determined, and pre-caching is carried out in advance. Thirdly, a comprehensive edge node selection scheme is constructed in combination with the adaptation degree, the received signal strength and the load rate; and then, the network control center screens out the edge node with the highest score for the vehicle terminal, and finally, the vehicle terminal selects the edge node with the highest score for access.

Description

Technical Field

[0001] The present invention belongs to the field of mobile communications, and in particular to an edge node selection scheme in an edge cache of an Internet of Vehicles. Background Art

[0002] With the rapid development of Internet of Vehicles technology, passengers' demand for riding experience is increasing, prompting smart cars to continuously innovate and provide diversified services. In order to achieve efficient content delivery, edge caching technology came into being. It stores service content on edge nodes closer to vehicle users, effectively shortening the content transmission path. It is worth noting that the delay for users to obtain content is not only related to whether the edge node caches the vehicle's request content, but also has a direct connection with the edge node selected by the vehicle to connect, such as the load rate of the edge node and the signal strength that the vehicle can receive. Therefore, when the vehicle selects an edge node to connect to obtain content, in addition to considering whether the edge node caches the vehicle's request content, it should also consider whether the current edge node's network conditions support transmission. The combination of the two can not only reduce the content acquisition delay, but also ensure the stability of the transmission process.

[0003] At present, the research on edge caching and node selection strategies for connected vehicles has made a lot of progress, but the existing solutions mostly focus on the optimization of a single dimension and fail to integrate the two. In the literature [Kazmi S MA, Dang TN, Yaqoob I, et al. Infotainment enabled smart cars: A joint communication, caching, and computation approach [J]. IEEE Transactions on Vehicular Technology, 2019, 68 (9): 8408-8420.], the authors jointly optimized cache resources and computing resources to minimize the delay of smart cars in obtaining entertainment content. In the literature [Mahmood A, Casetti C, Chiasserini CF, et al. Mobility-aware edge caching for connected cars [C] / / 2016 12th Annual Conference on Wireless On-demand Network Systems and Services (WONS). IEEE, 2016: 1-8.], the mobility of vehicles and the service time of edge nodes are deeply considered, and a mobility-aware edge caching strategy is proposed to ensure that the content is cached on the edge nodes that connected vehicles need to access most. In the above edge cache research, only the placement of cache content is considered, and the impact of network transmission conditions on content acquisition is not fully considered. Therefore, when the network conditions are poor, it may lead to a longer content acquisition delay. In the study of node selection schemes, the literature [Rahman MA, Salih QM, Asyhari AT, et al. Traveling distance estimation to mitigate unnecessary handoff in mobile wireless networks [J]. Annals of Telecommunications, 2019, 74: 717-726.] proposed to select appropriate edge nodes based on the received signal strength.The literature [Guo X, Omar MH, Zaini KM, et al. Access-Selection Algorithm for Heterogeneous Wireless Networks Based on Uncertain Network Attribute Values ​​[J]. Computational Intelligence and Neuroscience, 2022, 2022 (1): 4646889.] proposed to select edge nodes by obtaining the score of each edge node based on the received signal strength, network load and user rate as input decision parameters. However, the above node selection scheme research does not take into account the impact of cache content placement on content transmission, which may result in a longer transmission path and thus a longer content acquisition delay.

[0004] In view of the above shortcomings, the present invention proposes an edge node selection scheme that takes content placement into consideration. It realizes the effective coordination of edge caching and node selection, and allows user vehicles to obtain cached content from edge nodes as much as possible while ensuring stable network transmission conditions, shortening the content transmission path, reducing content acquisition latency, and improving user service quality. Summary of the invention

[0005] The present invention aims to solve the above problems of the prior art. An edge node selection scheme considering cache placement is proposed. The technical scheme of the present invention is as follows:

[0006] An edge node selection scheme considering cache placement in an Internet of Vehicles edge cache includes the following steps:

[0007] First, the network control center collects vehicle and network data on the edge node side composed of base stations and edge servers. Second, based on the collected data, the cache strategy matrix is ​​solved, the cache scheme is determined, and pre-caching is performed in advance. Third, a comprehensive edge node selection scheme is constructed by combining the adaptability, received signal strength, and load rate. Next, the network control center selects the edge node with the highest score for the vehicle terminal. Finally, the vehicle terminal selects the edge node with the highest score for access.

[0008] Further, the cache strategy matrix is ​​M=(m r,f ) R×F ,in represents the set of edge nodes, Represents a collection of content libraries. r,f =1 indicates that the content f has been cached on the edge node r, and 0 indicates that the content f has not been cached on the edge node r.

[0009] Further solving the cache strategy matrix specifically includes:

[0010] Content popularity definition:

[0011]

[0012] pl f represents the probability that content f is requested, where O f It represents the ranking of content f in descending order of requests, and 0<θ≤1 is the skewness factor.

[0013] Cache hit ratio definition:

[0014]

[0015] Further, the solution of the cache strategy matrix can be transformed into the problem of maximizing the regional cache hit rate, which includes:

[0016]

[0017] where ω f Indicates the size of the content f, C r represents the cache capacity of edge node r, constraint (4) represents the integer property of the cache decision, and constraint (5) indicates that the content cached by each edge node cannot exceed its maximum cache capacity.

[0018] Furthermore, the present invention solves problem (3) by using a reinforcement learning method. In order to effectively describe and simplify the modeling process of reinforcement learning, the present invention models the reinforcement learning process as a Markov decision process.

[0019] The further Markov decision process specifically includes:

[0020] (1) State space S:

[0021] M R×F (t): represents the cache state matrix at time t;

[0022] C r (t): represents the remaining cache space size of edge node r at time t.

[0023] Then the state s(t)∈S of the entire system at time t is:

[0024] s(t)=(M R×F (t),C r (t)) (6)

[0025] (2) The action space A represents the set of actions that can be taken in each state, mainly whether to cache the content f on the edge node r. Therefore, the cache action space a(t)∈A at time t is:

[0026] a(t)={m r,f (t)|m r,f (t)∈{0,1}} (7)

[0027] (3) The reward function is used to evaluate the pros and cons of taking a certain action and adjust the cache strategy by maximizing the cumulative expected reward. Given that the optimization problem is to maximize the cache hit rate, the reward function is defined as an indicator directly related to the hit rate:

[0028] rd(s(t),a(t))=HR(s(t),a(t)) (8)

[0029] There are many mature reinforcement learning algorithms. Among them, the reinforcement learning method based on proximal policy optimization uses clipping technology when updating the strategy to limit the strategy update range and avoid excessive strategy changes, making it more stable and easy to converge. It is suitable for various tasks. Based on the above advantages, this patent uses a reinforcement learning algorithm based on proximal policy optimization to solve the above optimization problem.

[0030] Further reinforcement learning algorithms based on proximal optimization strategies include:

[0031] Advantage function:

[0032] AF(s(t),a(t))=δ(t)+(γλ)δ(t)+...+(γλ) T-t+1 δ(T-1),γ∈(0,1),λ∈(0,1) (9)

[0033] A measure of how good an action is in a given state relative to the average action, where γ is the discount factor, λ is the hyperparameter, and δ(t) = r(t) + γV φ (s(t+1))-V φ (s(t+1)), s(t)∈S is used to represent the difference between the sum of the actual reward at time t plus the expected future reward and the estimated reward at the current time, V φ (s(t)) is the estimated value of the value network at time t.

[0034] Policy network θ:

[0035]

[0036] An action will be executed, and then the corresponding cache strategy will be adopted according to the action, where Γ represents the degree of change of the new cache strategy relative to the old cache strategy on the cache content item. When AF(s(t), a(t))>0, it means that the performance of selecting action a(t) under state s(t) is better than the average level. At this time, try to update the policy network in the direction of increasing the execution probability of action a(t). AF(s(t), a(t))<0 means that the performance of selecting action a(t) under state s(t) is not as good as the average level. At this time, try to update the policy network in the direction of reducing the execution probability of action a(t). In order to balance the trade-off between exploring new cache strategies and using already discovered cache strategies, the update range of the cache strategy is limited to (1-, 1+) ​​through the clip function, where ∈ is the clipping factor.

[0037] Value network φ:

[0038]

[0039] It is used to evaluate the quality of the current action taken by the policy network and update the policy network in a more valuable direction.

[0040] Furthermore, when the policy network θ and the value network φ tend to be stable, the cache state matrix M R×F Will also be determined.

[0041] The further definitions of adaptability, received signal strength, and load rate specifically include:

[0042] Adaptability: It is used to measure the degree to which the edge node caches the content requested by the vehicle. It is related to the cache strategy matrix. The greater the adaptability, the more content that the vehicle may request is cached by this edge node, that is, the more adapted the vehicle is to this edge node, and vice versa.

[0043] Received signal strength: It is a basic indicator for vehicle terminals to evaluate edge nodes, reflecting the channel quality of the network. The stronger the received signal strength, the more stable the channel.

[0044] Load rate: It is used to measure the ratio of the number of used resource blocks to the total number of resource blocks within the coverage area of ​​the edge node. The lower the load rate, the more resources can be allocated.

[0045] The further calculation of the adaptability, received signal strength, and load rate specifically includes:

[0046] Adaptability:

[0047] ad k,r =pl f m r,f (12)

[0048] represents the fitness of vehicle k relative to edge node r, where pl f represents the probability of vehicle request content f, m r,f Indicates whether edge node r has cached content f.

[0049] Received signal strength:

[0050]

[0051] represents the received signal strength between vehicle k and edge node r, where P r represents the wireless signal transmission power of edge node r, η represents the path loss factor, d k,r represents the distance from vehicle k to edge node r, X σ It represents Gaussian white noise with mean 0 and variance σ. Represents a collection of vehicles.

[0052] Load rate:

[0053]

[0054] It represents the ratio of the number of resource blocks used within the network coverage formed by edge node r to the total number of resource blocks, where U r is the total number of vehicles connected to the edge node r, h k,r is the number of resource blocks allocated by edge node r to the kth vehicle, H r is the total number of resource blocks that can be allocated to edge node r. In this paper, if the number of vehicles connected to the edge node is below the rated number, each vehicle is allocated a fixed number of resource blocks. When the number exceeds the rated number, each vehicle equally shares the total network resources.

[0055] Furthermore, in fuzzy decision-making, in order to deal with fuzzy parameter information, it is usually necessary to fuzzify these parameters, and the membership function is the conversion function between quantitative parameters and qualitative concepts. In this paper, the membership function maps the input to the membership on the [0,1] interval through the Mamdani method, indicating the degree to which the parameter belongs to this fuzzy set. The larger the membership value, the higher the degree to which the parameter belongs to this set. It mainly includes the following three steps:

[0056] 1) Fuzzification: Select the three parameters of adaptability, received signal strength and load rate as the three input parameters of the fuzzy logic system.

[0057] 2) Fuzzy reasoning: Fuzzy input and output are associated through fuzzy rules. Each input and output parameter is defined with three fuzzy logic levels, which are divided into low, medium and high. Therefore, there are 27 rules in the fuzzy rule base, that is, there are 27 possible output parameters. In order to ensure that the vehicle can select the most suitable edge node and distinguish the importance of the three parameters, the construction principle of the edge node load rate being the most important, followed by the received signal strength (RSS), and then the fitness parameter is constructed in descending order when designing the reasoning rules. In other words, the load rate has the most significant impact on the reasoning result, followed by RSS, and the fitness parameter is relatively less important. This design can more effectively reflect the different degrees of influence of edge node load, communication quality and fitness on the decision-making process.

[0058] 3) Defuzzification: The score of the edge node is the output of the fuzzy inference, and the output value is defined as three linguistic variables. Through the membership function, these linguistic variables are defuzzified so that the value of the edge node score is normalized between 0 and 1. The comprehensive output value after defuzzification is the score Y of the edge node r. r Finally, the edge node Y with the highest score is selected r To connect, that is:

[0059] Y r =Max{Y 1 ,Y 2 ,...,Y R} (15)

[0060] The advantages and beneficial effects of the present invention are as follows:

[0061] 1. In the research on edge caching, the impact of network transmission conditions on content transmission is fully considered, effectively reducing the content transmission delay caused by weak signal strength and high load rate.

[0062] 2. The adaptation parameter is introduced to achieve effective coordination between edge caching and node selection, thereby improving content transmission efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 is a network scene graph;

[0064] Figure 2 is the fitness membership function graph;

[0065] Figure 3 Load factor membership function diagram;

[0066] Figure 4 is the membership function diagram of received signal strength;

[0067] Figure 5is the score membership function graph;

[0068] Figure 6 This is the relationship between the average delay and the number of vehicles;

[0069] Figure 7 It is the relationship diagram between total delay and number of vehicles;

[0070] Figure 8 It is the relationship diagram between the blocking rate and the number of vehicles;

[0071] Fig. 9 This is a graph showing the relationship between cache hit rate and training rounds;

[0072] Fig.10 This is a graph showing the relationship between cache hit rate and cache capacity;

[0073] Fig.11 This is a graph showing the relationship between cache hit rate and the number of contents;

[0074] Fig.12 A summary diagram of the edge node selection scheme is provided; DETAILED DESCRIPTION

[0075] The following will describe the technical solutions in the embodiments of the present invention in detail in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention.

[0076] The technical solution of the present invention is as follows:

[0077] In the edge cache of the Internet of Vehicles, from the perspective of ensuring network transmission quality and shortening the content transmission path, an edge node selection scheme that considers content placement is designed, so that the edge node selected by the vehicle user can cache the content requested by the user as much as possible, reducing the content acquisition latency. At the same time, the impact of network conditions on content transmission is considered to avoid an unstable network transmission environment caused by weak received signal strength and high edge node load rate.

[0078] The edge node selection scheme considering content placement proposed in the present invention includes the following steps:

[0079] Step 1: Based on the historical request data collected by the edge nodes, the network control center uses the reinforcement learning method to calculate the cache strategy matrix, and caches the corresponding content on the edge nodes in advance according to the cache strategy matrix.

[0080] Step 2: The network control center collects terminal and network data, such as the location information of the vehicle terminal and the number of vehicles connected to the base station. Secondly, based on the collected data and the solved cache strategy matrix, the vehicle's received signal strength, adaptability and edge node load rate are calculated. Then, the received signal strength, adaptability and load rate are used as input to construct a fuzzy reasoning system to obtain the score of the edge node. Finally, the vehicle terminal selects the edge node with the highest score for access.

[0081] In order to evaluate the performance of the scheme proposed in this paper, a simulation platform was built using Python 3.9, Pytroch 2.1 and scikit-fuzzy 0.4.2 to simulate a two-way 4-lane urban core area at a 2km long intersection. It is assumed that 5 5G base stations are deployed, each equipped with an edge server, and the two together provide content services for vehicles. In most edge caching studies, it is usually assumed that the vehicle is within the coverage of a single edge node and is connected to the node by default. In this case, the basis for the vehicle to select the edge node is mainly the received signal strength, that is, the node selection problem is not considered. Therefore, in order to verify the effectiveness of the scheme in this paper, the comparison schemes in this paper are mainly: (1) Directly cache in advance through the least recently used scheme (Least Recently Used, LRU), the way the vehicle selects the edge node is to select only based on the received signal strength, recorded as the LR scheme. (2) The algorithm for solving the cache strategy matrix is ​​a learning algorithm based on Q-learning, and the way the vehicle selects the edge node is to select only based on the received signal strength, recorded as the QR scheme. (3) The algorithm for solving the cache strategy matrix is ​​a PPO-based learning algorithm. The way vehicles select edge nodes is based only on the received signal strength, which is denoted as the PR scheme.

[0082] Figure 7It reflects the relationship between the number of vehicles and the average delay. It can be seen that when the number of vehicles is 0-60, the system delay is mainly determined by whether the edge node connected to the vehicle caches the content requested by the vehicle. Since the edge node selection scheme proposed by the present invention comprehensively considers the parameter of fitness, when the network conditions are good, the vehicle can give priority to connecting to the edge node with higher fitness, and directly obtain the requested content from the cache with a greater probability, without having to access it through the remote data center, thereby significantly reducing the content acquisition delay. For the other three schemes (QR, PR, LR), since the parameter of fitness is not considered, whether the vehicle can directly obtain the cache content depends on the cache strategy scheme. Compared with the Q-learning learning algorithm, since the PPO learning algorithm adopts proximal strategy optimization in the solution process, the strategy update range is limited, so that the PPO algorithm can adjust the cache content more flexibly and better meet the dynamic needs of vehicle users. Therefore, when the number of vehicles is small and the network conditions are good, the delay generated by connecting to the edge node through the PR scheme to obtain content is lower than that of the QR scheme. On the contrary, compared with the above learning algorithms, the LRU cache scheme cannot capture the dynamic changes of user requests and the access frequency of content in time, so that even under good network conditions, it still generates higher latency. As the number of vehicles gradually increases, the total latency of the four schemes shows an upward trend. However, the differences in load management among different schemes lead to significantly different latency growth characteristics. For the three schemes of QR, PR, and LR, since the load status of each node is not fully considered when selecting edge nodes, vehicles tend to connect to certain edge nodes in high-density areas, resulting in a sharp decline in the resource allocation efficiency of these nodes, and ultimately causing network congestion and increased latency. This load imbalance reduces the service capacity of edge nodes and affects the overall network performance. In contrast, the scheme proposed in this paper comprehensively evaluates the load level of each edge node, reasonably allocates the connection needs of vehicles, avoids the imbalance of load between nodes, and thus significantly alleviates the problem of network congestion. Specifically, by dynamically adjusting the connection strategy of edge nodes, the scheme in this paper can effectively disperse the connection of vehicles, ensure that the load of each edge node fluctuates within a reasonable range, and avoid the latency bottleneck caused by excessive resource use. Therefore, in the scenario with a large number of vehicles, the latency of the scheme proposed in this paper is always significantly lower than that of the other three algorithms, showing better scalability and stability.

[0083] Figure 8 The relationship between the total delay and the number of vehicles is shown. Figure 7Similarly, when the number of vehicles is between 0 and 60, the total delay of the proposed solution is lower than that of the other three solutions. As the number of vehicles increases, the total delay of the four solutions increases. Since the proposed solution takes the load rate into consideration, network congestion is avoided, and the total delay is lower than that of the other three solutions. Figure 7 and Figure 8 It can be seen that relying solely on the cache placement scheme and ignoring the network conditions of edge nodes may cause a large delay when vehicles obtain content as the number of vehicles increases. Therefore, considering network conditions is crucial to optimizing system performance.

[0084] Fig. 9 Reflecting the relationship between the number of vehicles and the blocking rate, when the number of vehicles is less than 40, the blocking rates of the four schemes are all maintained at 0% because the maximum number of vehicles that a single edge node can serve is not exceeded. However, when the number of vehicles exceeds 40, as the vehicle density increases, the blocking rates of the other three schemes begin to gradually increase. This is mainly because when vehicles select edge nodes, they only rely on the received signal strength (RSS) as a basis for judgment, without fully considering the actual load of the edge nodes. This leads to the tendency of them to choose the same edge node with the strongest RSS for connection when a large number of vehicles are concentrated in a certain area, thereby exceeding the service capacity limit of the node and causing the blocking rate to increase. In contrast, the scheme proposed in this paper introduces the edge node load rate as one of the selection criteria, and can try to select edge nodes with lower load rates each time the connection target is selected. This strategy effectively disperses the request pressure of vehicles on edge nodes and avoids the blocking problem caused by overloading of a single node. Therefore, when the number of vehicles increases, the blocking rate of the algorithm in this paper remains at a lower level than that of the other three algorithms.

[0085] Fig.10 Reflecting the relationship between the change of cache hit rate and the number of training rounds, in the first 200 training rounds, due to sufficient cache space, the edge node can continue to cache the video content frequently requested by vehicle users, resulting in a gradual increase in the cache hit rate. However, around the 200th round, the cache space tends to be saturated and no more content can be cached, causing the cache hit rate to stabilize. Since the edge node selection scheme proposed in this paper introduces the fitness parameter, under good network conditions, the key factor affecting the cache hit rate is whether the edge node to which the vehicle is connected has cached its requested content. By considering the fitness parameter, the scheme proposed in this paper makes it more likely that the vehicle will connect to the edge node that caches its requested content. Therefore, compared with the other three schemes, the scheme proposed in this paper has obvious advantages in cache hit rate.

[0086] Fig.11This reflects the trend of the cache hit rate changing with the size of the cache space. As the cache space increases, the edge node can cache more video content requested by users, so the cache hit rates of the four solutions increase with the increase of the cache space. Fig.12 The cache hit rate reflects the trend of changes in the size of the content library. As the number of contents increases, the number of user-requested contents in the edge nodes decreases, which leads to a decrease in the cache hit rate of the four schemes as the number of contents increases. Since the scheme in this paper takes the parameter of adaptability into consideration, the cache hit rate of the scheme in this paper is higher than that of the other three algorithms.

[0087] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0088] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0089] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0090] The above embodiments should be understood to be only used to illustrate the present invention and not to limit the protection scope of the present invention. After reading the contents of the present invention, technicians can make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. An edge node selection scheme considering content placement, characterized in that: The following steps are involved:

101. At the edge node side composed of the base station and the edge server, the vehicle and network data are collected through the network control center to solve the cache strategy matrix and pre-cache in advance; 102. According to the cache strategy matrix obtained in step 101 and the collected network data, the network control center calculates the compatibility, received signal strength and load rate between the vehicle and the edge node; 103. Construct a comprehensive edge node selection scheme according to the adaptability, load rate, and received signal strength obtained in step 102; 104. According to the edge node scores obtained in step 103, the vehicle terminal selects the edge node with the highest score for access.

2. An edge node selection scheme considering content placement according to claim 1, characterized in that Based on the collected vehicle request data, the cache strategy matrix is ​​solved, including: The cache strategy matrix is ​​M = (m r,f ) R×F ,in represents the set of edge nodes, Represents a collection of content libraries. r,f =1 indicates that the content f has been cached on the edge node r, and 0 indicates that the content f has not been cached on the edge node r. Content popularity definition: pl f represents the probability that content f is requested, where O f It represents the ranking of content f in descending order of requests, and 0<θ≤1 is the skewness factor. Cache hit ratio definition: Solving the cache strategy matrix is ​​transformed into the problem of maximizing the regional cache hit rate, which includes: where ω f Indicates the size of the content f, C r represents the cache capacity of edge node r, constraint (4) represents the integer property of the cache decision, and constraint (5) indicates that the content cached by each edge node cannot exceed its maximum cache capacity. The reinforcement learning method is used to solve problem (3). In order to effectively describe and simplify the modeling process of reinforcement learning, the present invention models the reinforcement learning process as a Markov decision process. The Markov decision process specifically includes: (1) State space S: M R×F (t): represents the cache state matrix at time t; C r (t): represents the remaining cache space size of edge node r at time t. Then the state s(t)∈S of the entire system at time t is: s(t)=(M R×F (t),C r (t)) (6) (2) The action space A represents the set of actions that can be taken in each state, mainly whether to cache the content f on the edge node r. Therefore, the cache action space a(t)∈A at time t is: a(t)={m r,f (t)|m r,f (t)∈{0,1}} (7) (3) The value of the reward function is used to evaluate the pros and cons of taking a certain action and adjust the strategy by maximizing the cumulative expected reward. Given that the optimization problem is to maximize the cache hit rate, the reward function is defined as an indicator directly related to the hit rate: rd(s(t),a(t))=HR(s(t),a(t)) (8) There are many mature reinforcement learning algorithms. Among them, the reinforcement learning method based on proximal policy optimization uses clipping technology when updating the strategy to limit the strategy update range and avoid excessive strategy changes, making it more stable and easy to converge. It is suitable for various tasks. Based on the above advantages, this patent uses a reinforcement learning algorithm based on proximal policy optimization to solve the above optimization problem. The reinforcement learning algorithms based on proximal optimization strategies specifically include: Advantage function: AF(s(t),a(t))=δ(t)+(γλ)δ(t)+...+(γλ) T-t+1 δ(T-1),γ∈(0,1),λ∈(0,1) (9) A measure of how good an action is in a given state relative to the average action, where γ is the discount factor, λ is the hyperparameter, and δ(t) = r(t) + γV φ (s(t+1))-V φ (s(t+1)), s(t)∈S is used to represent the difference between the sum of the actual reward at time t plus the expected future reward and the estimated reward at the current time, V φ (s(t)) is the estimated value of the value network at time t. Policy network θ: An action will be executed, and then the corresponding cache strategy will be adopted according to the action, where Γ represents the degree of change of the new cache strategy relative to the old cache strategy on the cache content item. When AF(s(t), a(t))>0, it means that the performance of selecting action a(t) under state s(t) is better than the average level. At this time, try to update the policy network in the direction of increasing the execution probability of action a(t). AF(s(t), a(t))<0 means that the performance of selecting action a(t) under state s(t) is not as good as the average level. At this time, try to update the policy network in the direction of reducing the execution probability of action a(t). In order to balance the trade-off between exploring new cache strategies and using already discovered cache strategies, the update range of the cache strategy is limited to (1-∈,,1+∈) through the clip function, where ∈ is the clipping factor. Value network φ: It is used to evaluate the quality of the current action taken by the policy network and update the policy network in a more valuable direction. When the policy network θ and the value network φ tend to be stable, the cache state matrix M R×F Will also be determined.

3. An edge node selection scheme considering content placement according to claim 1, characterized in that: The definitions of adaptability, received signal strength, and load rate based on the cache strategy matrix and the collected network data include: Adaptability: It is used to measure the degree to which the edge node caches the content requested by the vehicle. The greater the adaptability, the more content that the vehicle may request is cached by this edge node, that is, the more adapted the vehicle k is to the edge node r, and vice versa. Received signal strength: It is a basic indicator for vehicle terminals to evaluate edge nodes, reflecting the channel quality of the network. The stronger the received signal strength, the more stable the channel. Load rate: It is used to measure the ratio of the number of used resource blocks to the total number of resource blocks within the coverage area of ​​the edge node. The lower the load rate, the more resources can be allocated.

4. An edge node selection scheme considering content placement according to claim 3, characterized in that: The calculation of adaptability, received signal strength, and load rate specifically includes: Adaptability: ad k,r =pl f m r,f (12) represents the fitness of vehicle k relative to edge node r, where pl f represents the probability of vehicle request content f, m r,f Indicates whether edge node r has cached content f. Received signal strength: represents the received signal strength between vehicle k and edge node r, where P r represents the wireless signal transmission power of edge node r, η represents the path loss factor, d k,r represents the distance from vehicle k to edge node r, X σ represents Gaussian white noise with a mean of 0 and a variance of σ, where Represents a collection of vehicles. Load rate: It represents the ratio of the number of resource blocks used within the network coverage formed by edge node r to the total number of resource blocks, where U r is the total number of vehicles connected to the edge node r, h k,r is the number of resource blocks allocated by edge node r to the kth vehicle, H r is the total number of resource blocks that can be allocated to edge node r. In this paper, if the number of vehicles connected to the edge node is below the rated number, each vehicle is allocated a fixed number of resource blocks. When the number exceeds the rated number, each vehicle equally shares the total network resources. An edge node selection scheme considering content placement according to claim 4, characterized in that: In fuzzy decision-making, in order to deal with fuzzy parameter information, it is usually necessary to fuzzify these parameters, and the membership function is the conversion function between quantitative parameters and qualitative concepts. In this paper, the membership function maps the input to the membership on the [0,1] interval through the Mamdani method, indicating the degree to which the parameter belongs to this fuzzy set. The larger the membership value, the higher the degree to which the parameter belongs to this set. It mainly includes the following three steps: 1) Fuzzification: Select the three parameters of adaptability, received signal strength and load rate as the three input parameters of the fuzzy logic system. 2) Fuzzy reasoning: Fuzzy input and output are associated through fuzzy rules. Each input and output parameter is defined with three fuzzy logic levels, which are divided into low, medium and high. Therefore, there are 27 rules in the fuzzy rule base, that is, there are 27 possible output parameters. In order to ensure that the vehicle can select the most suitable edge node and distinguish the importance of the three parameters, the construction principle of the edge node load rate being the most important, followed by the received signal strength (RSS), and then the fitness parameter is constructed in descending order when designing the reasoning rules. In other words, the load rate has the most significant impact on the reasoning result, followed by RSS, and the fitness parameter is relatively less important. This design can more effectively reflect the different degrees of influence of edge node load, communication quality and fitness on the decision-making process. 3) Defuzzification: The score of the edge node is the output of the fuzzy inference, and the output value is defined as three linguistic variables. Through the membership function, these linguistic variables are defuzzified so that the score value of the edge node is normalized between 0 and 1. The comprehensive output value after defuzzification is the score of the edge node r. r Finally, the edge node Y with the highest score is selected r To connect, that is: AND r =Max{Y1,Y2,...,Y R } (15)。

Images