A method and system based on online joint recommendation and edge-assisted caching

Abstract

The present invention discloses a method and system based on online joint recommendation and edge-assisted caching, the method comprising: setting the original state characteristics of an intelligent agent based on task requirements; embedding the original state characteristics of the intelligent agent based on a graph convolutional network to obtain a graph embedding of the intelligent agent; inputting the graph embedding of the intelligent agent into an actor-critic network to obtain the recommended action, cached action and reward of the intelligent agent; and taking the recommended action and cached action corresponding to the maximum reward as decision outputs based on the reward maximization principle. The system comprises an original state setting module, a graph embedding module, a reinforcement learning module and a decision output module. By using the present invention, the diversified needs and dynamic network conditions of mobile phone users can be captured, and the problems of low recommendation accuracy and slow loading speed that exist when mobile phone users obtain online video content can be solved. The present invention can be widely applied to the field of reinforcement learning technology.

Description

Technical Field

[0001] The present invention relates to the technical field of reinforcement learning, and in particular to a method and system based on online joint recommendation and edge-assisted caching. Background Art

[0002] Online video recommendations and edge-assisted caching for mobile phone users are currently hot topics in the research field. In recent years, a large number of studies on video cache optimization have emerged in the field of short video streaming to improve the quality and efficiency of video transmission. However, existing research mainly focuses on caching popular video content, ignoring the importance of dynamic personalized recommendations based on the personal preferences of mobile phone users.

[0003] Secondly, graph neural networks have been widely used and studied in video recommendation, by modeling the relationship between users and videos to improve the accuracy of personalized recommendations. Although technologies such as graph neural networks have shown good performance in personalized recommendations, most recommendation studies ignore the impact of network conditions on user quality of experience (QoE). When the video content in the recommendation list is not cached on the road side unit (RSU) or mobile phone device, directly retrieving the video content from the cloud will result in significant transmission delays. Although these recommended videos meet user preferences, high latency will cause buffering problems during playback, weakening the effect of the recommendation. Therefore, the existing recommendation system cannot fully optimize the user's viewing experience, resulting in low-quality video content during video playback, affecting user satisfaction and stickiness.

[0004] Existing research focuses on recommendation systems and video cache optimization independently, without considering the impact of user preferences and network conditions on the quality of experience (QoE) of video streaming. This leads to some challenges in practical applications, such as users may receive video recommendations that do not match their interests in scenarios that only focus on cache, or users may experience high latency and low quality during video playback in scenarios that only focus on video recommendations. Summary of the invention

[0005] In order to solve the above technical problems, the purpose of the present invention is to provide a method and system based on online joint recommendation and edge-assisted caching, which can capture the diverse needs and dynamic network conditions of mobile phone users and solve the problems of low recommendation accuracy and slow loading speed when mobile phone users obtain online video content.

[0006] The first technical solution adopted by the present invention is: a method based on online joint recommendation and edge-assisted caching, comprising the following steps:

[0007] Set the original state characteristics of the agent based on task requirements;

[0008] Embedding the original state features of the agent based on a graph convolutional network to obtain a graph embedding of the agent;

[0009] Input the agent’s graph embedding into an actor-critic network to obtain the agent’s recommended actions, cached actions, and rewards;

[0010] Based on the reward maximization principle, the recommended action and cache action corresponding to the maximum reward are used as decision outputs.

[0011] Furthermore, the original state characteristics of the agent include video cache information, video recommendation information and coordinate information.

[0012] Furthermore, the step of embedding the original state features of the agent based on the graph convolutional network to obtain the graph embedding of the agent specifically includes:

[0013] Based on the graph convolutional network, the environment is designed as a graph, and the intelligent agent is used as a node on the graph;

[0014] The nodes are connected based on the coverage of the agent, and the original state features of the agent are transmitted to the connected nodes along the connection line to obtain the propagation model of the graph convolutional network;

[0015] The interconnected original state features are used as the input features of the agent, and the input features of the agent are subjected to several layers of graph message passing based on the propagation model of the graph convolutional network to obtain the graph embedding of the agent.

[0016] Through this preferred step, it is possible to capture the heterogeneity between mobile phone users and roadside units, and capture the dynamic relationships caused by the movement of mobile phone users, compress the original state features and reveal the potential dynamic environment patterns.

[0017] Furthermore, the recommended actions and cached actions of the agent are expressed as follows:

[0018]

[0019] Among them, a n (t) represents the action space of the agent; Represents the roadside unit r at time period t i The bitrate of cached content k; represents the number of mobile phone users v in time period t j The bitrate of cached content k; Indicates the video content k in the time period for mobile phone users v j Whether it is recommended; K represents the total set of video content k.

[0020] Furthermore, the reward of the agent is expressed as follows:

[0021]

[0022]

[0023]

[0024]

[0025]

[0026] Among them, φ(t) represents the reward of the agent and also represents the quality experience of mobile phone users; f t (v j , k) represents the number of mobile phone users v j and video content k as the recommendation score of the input; Indicates the video content k in the mobile phone user v at time period t j whether it is recommended; N represents the total set of roadside units; M represents the total set of mobile phone users; K represents the total set of video content k; Indicates the transmission of video content k to mobile phone user v j Total delay of Represents mobile phone users v j The nearest roadside unit at time t; Represents video content k from the nearest roadside unit Transmit to mobile phone usersv j transmission delay; represents the video content k transmitted from the main base station to the nearest roadside unit Delays; represents the number of mobile phone users v in time period t j The bitrate of cached content k; Indicates the nearest roadside unit at time period t-1 The bitrate of cached content k; α represents the coefficient that captures the relative importance of the recommendation score and transmission delay; Indicates whether transmission delay will occur; k Indicates the size of each video content; represents the delay of video content k being obtained from a neighboring roadside unit or from the main base station; i′ represents the number of the neighboring roadside unit; r i ′ represents the neighbor roadside unit; Represents the roadside unit r i 'Whether the required video content k is cached; Indicates whether to obtain video content k from neighboring roadside unit i′ to mobile phone user v j ; Indicates whether to obtain video content k from the main base station to the mobile phone user v j ; B represents the main base station; Indicates that video content k is transmitted from roadside unit i′ to the nearest roadside unit transmission delay; represents the video content k transmitted from the main base station to the nearest roadside unit transmission delay.

[0027] Furthermore, the transmission delay of the video content is affected by the transmission rate of the channel, and the expression of the transmission rate of the channel is as follows:

[0028]

[0029]

[0030]

[0031]

[0032]

[0033] R Back (t) = min{R MBS (t), R CN (t)}

[0034] in, represents the number of mobile phone users v in time period t j and roadside units i The transmission rate between Represents the roadside unit r i The channel bandwidth allocated to mobile phone users; Represents the roadside unit r i The signal transmission power of 2 represents the noise power; Indicates the time period t from the roadside unit r i To mobile phone users i The channel gain; ∈ represents the path loss coefficient; Represents the roadside unit r i The antenna gain factor; Represents mobile phone users v j With roadside unit i The distance between them; h0(t) represents the Rayleigh channel fading random variable that obeys the complex Gaussian distribution, and its value range is between 0 and 1; MBS represents the main base station; Represents the roadside unit r in time period t i and the transmission rate between the main base station; B0 represents the channel bandwidth allocated to the roadside unit by the main base station; P bs Indicates the signal transmission power of the primary base station; Represents the distance from the main base station to the roadside unit r in time period t i The channel gain of G bs Indicates the antenna gain coefficient of the primary base station; Represents the main base station and roadside unit r i The distance between CN (t) represents the transmission delay from the core network to the main base station; C MBS Indicates the backhaul bandwidth when the number of roadside units connected to the main base station is less than or equal to the number of roadside units connected to the main base station; C CN represents the backhaul bandwidth when the number of roadside units connected to the main base station is greater than the number of roadside units connected to the main base station; N(t) represents the number of roadside units connected to the main base station in time period t; N Max Indicates the maximum number of roadside units carried by the main base station; R Back (t) represents the backhaul bandwidth from the core network to the roadside unit; R MBS (t) represents the backhaul bandwidth from the main base station to the roadside unit; R CN (t) represents the backhaul bandwidth from the core network to the master base station.

[0035] Furthermore, when the number of roadside units without cached video content is higher than a set critical threshold, the roadside unit and main base station with video content with the smallest delay estimated from historical records are selected for video content recommendation and caching.

[0036] The second technical solution adopted by the present invention is: a system based on online joint recommendation and edge-assisted caching, comprising:

[0037] The original state setting module sets the original state characteristics of the agent based on task requirements;

[0038] A graph embedding module, which embeds the original state features of the agent based on a graph convolutional network to obtain a graph embedding of the agent;

[0039] A reinforcement learning module for inputting the agent's graph embedding into an actor-critic network to obtain the agent's recommended actions, cached actions, and rewards;

[0040] The decision output module takes the recommended action and cached action corresponding to the maximum reward as the decision output based on the reward maximization principle.

[0041] The beneficial effects of the method and system of the present invention are as follows: the present invention captures the heterogeneity between mobile phone users and roadside units by means of graph convolutional network embedding, and captures the dynamic relationship generated by the movement of mobile phone users; constructs a multi-agent reinforcement learning framework based on the actor-critic architecture, inputs the graph embedded with the state relationship representation into the multi-agent reinforcement learning framework for learning optimization, outputs recommendations and cache actions, and formulates learning rewards in combination with recommendation scores and transmission costs; ultimately achieves the capture of the diverse needs and dynamic network conditions of mobile phone users, and solves the problems of low recommendation accuracy and slow loading speed when mobile phone users obtain online video content. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a flow chart of steps of a method based on online joint recommendation and edge-assisted caching of the present invention;

[0043] Figure 2 It is a structural block diagram of a system based on online joint recommendation and edge-assisted caching of the present invention;

[0044] Figure 3 It is a schematic diagram of a framework of a method based on online joint recommendation and edge-assisted caching of the present invention;

[0045] Figure 4 This is a comparison chart of the quality experience effect of a method based on online joint recommendation and edge-assisted caching of the present invention and other benchmark algorithms;

[0046] Figure 5 It is a schematic diagram comparing the effects of a method based on online joint recommendation and edge-assisted caching of the present invention and a MAPPO algorithm without graph neural network;

[0047] Figure 6 It is a schematic diagram comparing the retrieval cost of a method based on online joint recommendation and edge-assisted caching of the present invention and the MAPPO algorithm;

[0048] Figure 7 It is a schematic diagram comparing recommendation scores of a method based on online joint recommendation and edge-assisted caching of the present invention and a collaborative filtering algorithm (CF);

[0049] Figure 8 It is a schematic diagram comparing the impact of the number of intelligent agents on a method based on online joint recommendation and edge-assisted caching of the present invention and an existing method;

[0050] Fig. 9 1. It is a schematic diagram comparing the impact of the number of video contents of a method based on online joint recommendation and edge-assisted caching of the present invention and an existing method;

[0051] Fig.10It is a schematic diagram of retrieval cost of a method based on online joint recommendation and edge-assisted caching of the present invention;

[0052] Fig.11 It is a schematic diagram of a recommendation index of a method based on online joint recommendation and edge-assisted caching of the present invention. DETAILED DESCRIPTION

[0053] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only provided for the convenience of explanation and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0054] Reference Figure 1 and Figure 3 The present invention provides a method based on online joint recommendation and edge-assisted caching, the method comprising the following steps:

[0055] S1. Set the original state characteristics of the agent based on task requirements;

[0056] Specifically, in the optimization scenario of the present invention, the joint optimization of mobile phone video caching and personalized recommendation must take into account the caching and recommendation requirements of each multi-edge access MEC and mobile phone user. This involves striking a balance between the minimum necessary and maximum available cache capacity, while adapting to the dynamic needs of different video content. Therefore, each agent n∈[M+N], where n represents the agent; N represents the total set of roadside units; M represents the total set of mobile phone users; must continuously monitor the status of connected MECs and mobile phone users, video content requests, and potential future caching and recommendation tasks. Therefore, the original state features that define the agent include video caching information of roadside units and mobile phone users, video recommendation information of roadside units and mobile phone users, x-axis coordinate information of roadside units and mobile phone users, and y-axis coordinate information of roadside units and mobile phone users, that is, s n，t ={c n，t , f n，t , x n，t ,y n，t}, where c n，t Represents the video cache information of the agent; f n，t Represents the video recommendation information of the agent; x n,t Represents the x-axis coordinate information of the intelligent agent; y n，t Represents the y-axis coordinate information of the agent.

[0057] Video cache informationc n，tis a vector representing the content status of different cached videos, which is a concatenation of the cached video type, cached video size, and the time elapsed since the last access. This feature set not only provides a snapshot of the current cache status, but also indicates the potential future popularity of different video types.

[0058] Video recommendation information n，t is a set of recommendation scores for various video contents observed, and the recommendation score is denoted as f t (v j , k), video recommendation information f n，t Each score in is derived from historical data using collaborative filtering techniques and updated by a graph convolutional network recommendation function, enabling the system to predict user preferences and needs for different videos. These scores are represented as vectors That is, there are K video contents associated with agent n, reflecting the expected user interactivity and satisfaction of each video content. It is crucial for the agent to make decisions about which videos to recommend, which is used to balance user preferences with the current state and goals of the system.

[0059] S2. embedding the original state features of the agent based on a graph convolutional network to obtain a graph embedding of the agent;

[0060] Specifically, in order to alleviate the curse of dimensionality, the present invention allows each agent n to observe only the original state features {c n，t , f n，t , x n，t ,y n,t}, but they are prone to fall into local optimal solutions. Therefore, it is necessary to globally represent the characteristics of all agents, but the present invention cannot simply represent the original state characteristics s of all agents n n，t Stacked into a global state matrix and provided to the model of each agent, the inherent dynamic graph structure between agents will be lost. If the state space is directly divided into subspaces for each individual agent, it will lead to lack of coordination, inefficiency, and failure to capture the heterogeneity between mobile phone users and RSUs, and more importantly, failure to capture their dynamic relationships due to the movement of mobile phone users. The present invention proposes to embed the entire graph into a neural network and allow iterative state interactions between agents, strengthening the information between agents over time according to importance. The generated state has a manageable dimension and incorporates useful states of other agents. The graph convolutional network is designed to embed link relationships and their semantics, which is very consistent with the state representation of the present invention. Therefore, the present invention adopts a graph convolutional network (GCN) with two heterogeneous subgraphs to abstract different nodes, and the output of the GCN is used as the abstract state of each agent, compressing the original state features and revealing potential dynamic environment patterns.

[0061] S2.1. Based on the graph convolutional network, the environment is designed as a graph and the agent is used as a node on the graph.

[0062] Specifically, the environment is designed as a graph in It is a set of agent nodes, including M+N agents, namely the current mobile phone users and roadside units.

[0063] S2.2, connect the nodes based on the coverage of the agent, and transfer the original state features of the agent along the connection line to the connected nodes to obtain the propagation model of the graph convolutional network;

[0064] Specifically, if the mobile phone user node v j ∈M in roadside unit r i ∈N, then there is a connection between them; similarly, when the roadside unit node r i ∈N is within each other’s coverage, it is connected to the adjacent roadside unit node r i′ ∈N connected. Information propagation in graph convolutional networks passes features as messages from neighbors to each node And aggregate them with the features of the agent. The propagation model of the graph convolutional network of the present invention is standardized as:

[0065]

[0066] in, Represents the input features of the agent node.

[0067] S2.3. The interconnected original state features are used as the input features of the agent, and the input features of the agent are subjected to several layers of graph message passing based on the propagation model of the graph convolutional network to obtain the graph embedding of the agent.

[0068] Specifically, the interconnected original state features are used as the input features of the agent. At this time, the initial global state input is expressed as As the node embedding propagates in each layer l, i.e. Where g(·) represents a two-level convolutional network that aggregates the features of each node with its neighbors. After L layers of graph message passing, the final agent graph embedding is obtained. Eventually Mapped to the corresponding agent as input to the actor network.

[0069] S3 inputs the graph embedding of the agent into the actor-critic network to obtain the recommended action, cached action and reward of the agent;

[0070] Specifically, in the system, the decision of each agent involves determining whether to perform recommendation or caching and the bit rate of cached video in the roadside unit and mobile phone user network. Each agent represents a roadside unit or a mobile phone user; there are two types of decisions represented as actions: if the agent controls a mobile phone device, the most appropriate recommended content is determined based on user preferences and network dynamics, and factors such as storage capacity and network transmission rate are considered to decide whether to cache specific video content and the storage bit rate for mobile phone users and RSU agents. Using the popular actor-critic framework, each agent in the present invention takes the graph convolutional network embedding of the roadside unit, mobile phone device and video where the mobile phone user is located as the input of its actor network, and outputs recommendations, caching actions and rewards.

[0071] The recommended actions and cached actions of the agent are expressed as follows:

[0072]

[0073] Among them, a n (t) represents the action space of the agent; Represents the roadside unit r at time period t i The bitrate of cached content k; represents the number of mobile phone users v in time period t j The bitrate of cached content k; Indicates the video content k in the time period for mobile phone users v j Whether it is recommended; K represents the total set of video content k.

[0074] In a specific embodiment of the present invention, the formulation of the reward is a comprehensive indicator that reflects the effectiveness of the video content caching and recommendation strategy. For each agent at a given time step t, the reward is equal to the quality experience QoE of the video stream, which combines two key performance indicators: recommendation score and transmission cost (also known as retrieval cost). The recommendation score is a positive value that quantifies the effectiveness of the video content recommendation made by the agent, and it will be rewarded for recommendations that accurately match user preferences and needs. In contrast, the retrieval cost is a penalty term that reflects the resources consumed to obtain and cache video content, including factors such as bandwidth usage and latency. This cost is deducted from the recommendation score, thereby encouraging the agent to strike a balance between recommending popular or high-demand video content and efficiently managing related costs.

[0075] The reward of the agent is expressed as follows:

[0076]

[0077]

[0078]

[0079]

[0080]

[0081] Among them, φ(t) represents the reward of the agent and also represents the quality experience of mobile phone users; f t (v j , k) represents the number of mobile phone users v j and video content k as the recommendation score of the input; Indicates the video content k in the mobile phone user v at time period t j whether it is recommended; N represents the total set of roadside units; M represents the total set of mobile phone users; K represents the total set of video content k; Indicates the transmission of video content k to mobile phone user v j Total delay of Represents mobile phone users v j The nearest roadside unit at time t; Represents video content k from the nearest roadside unit Transmit to mobile phone usersv j transmission delay; represents the video content k transmitted from the main base station to the nearest roadside unit Delays; represents the number of mobile phone users v in time period t j The bitrate of cached content k; Indicates the nearest roadside unit at time period t-1 The bitrate of cached content k; α represents the coefficient that captures the relative importance of the recommendation score and transmission delay; Indicates whether transmission delay will occur; k Indicates the size of each video content; represents the delay of video content k being obtained from a neighboring roadside unit or from the main base station; i′ represents the number of the neighboring roadside unit; r i′ represents a neighbor roadside unit; Represents the roadside unit r i′ Whether the required video content k is cached; Indicates whether to obtain video content k from neighboring roadside unit i′ to mobile phone user v j ; Indicates whether to obtain video content k from the main base station to the mobile phone user v j ; B represents the main base station; Indicates that video content k is transmitted from roadside unit i′ to the nearest roadside unit transmission delay; represents the video content k transmitted from the main base station to the nearest roadside unit transmission delay.

[0082] In the process of constructing the reward, the specific embodiment of the present invention assumes that each mobile phone device connected to the roadside unit or the main base station (MBS) is assigned an independent orthogonal channel. The transmission delay of the video content is affected by the transmission rate of the channel. According to Shannon's theorem, the expression of the transmission rate of the channel is as follows:

[0083]

[0084]

[0085]

[0086]

[0087]

[0088] R Back (t) = min{R MBS (t), R CN (t)}

[0089] in, represents the number of mobile phone users v in time period t j and roadside units i The transmission rate between Represents the roadside unit r i The channel bandwidth allocated to mobile phone users; Represents the roadside unit r i The signal transmission power of 2 represents the noise power; Indicates the time period t from the roadside unit r i To mobile phone users j The channel gain; ∈ represents the path loss coefficient; Represents the roadside unit r i The antenna gain factor; Represents mobile phone users v j With roadside unit i The distance between them; h0(t) represents the Rayleigh channel fading random variable that obeys the complex Gaussian distribution, and its value range is between 0 and 1; MBS represents the main base station; Represents the roadside unit r in time period t i and the transmission rate between the main base station; B0 represents the channel bandwidth allocated to the roadside unit by the main base station; P bs Indicates the signal transmission power of the primary base station; Represents the distance from the main base station to the roadside unit r in time period t i The channel gain of G bs Indicates the antenna gain coefficient of the primary base station; Represents the main base station and roadside unit r i The distance between CN (t) represents the transmission delay from the core network to the main base station; C MBS Indicates the backhaul bandwidth when the number of roadside units connected to the main base station is less than or equal to the number of roadside units connected to the main base station; C CN represents the backhaul bandwidth when the number of roadside units connected to the main base station is greater than the number of roadside units connected to the main base station; N(t) represents the number of roadside units connected to the main base station in time period t; N Max Indicates the maximum number of roadside units carried by the main base station; R Back (t) represents the backhaul bandwidth from the core network to the roadside unit; R MBS (t) represents the backhaul bandwidth from the main base station to the roadside unit; R CN (t) represents the backhaul bandwidth from the core network to the master base station.

[0090] Taking into account the actual network conditions and the impact of the number of transmissions from the main base station to the roadside units, the specific embodiments of the present invention found that many roadside units often access the video content in the main base station at the same time, especially during peak traffic hours and when the roadside unit storage is limited, which leads to network congestion and delays, and increases the dependence on the main base station. In addition, an undesirable result was observed in the embodiment: there are a large number of roadside units that choose not to cache content. In order to solve this problem, the present invention designs a rule-based method, which strategically intervenes when the number of roadside units that do not cache video content is higher than a set critical threshold, and selects the roadside units and main base stations with video content with the smallest delay estimated from historical records for video content recommendation and caching, that is,

[0091]

[0092] in, represents the delay of video content k being obtained from a neighboring roadside unit or from the main base station; Represents the roadside unit r i′ Whether the required video content k is cached; represents the video content k transmitted from the main base station to the nearest roadside unit transmission delay.

[0093] The number of roadside units that do not have cached video content is measured using the parameter Indicated by; the critical threshold is set by Q min express.

[0094] S4. Based on the reward maximization principle, the recommended action and cached action corresponding to the maximum reward are used as decision outputs.

[0095] In order to verify the effect of the method of the present invention, a comparative experiment was conducted between the method of the present invention and the prior art method.

[0096] Reference Figure 4 According to the user experience quality index of the method of the present invention, compared with the user experience quality index of the baseline method (random algorithm, greedy algorithm, DQN algorithm, MADDPG algorithm), it can be found that compared with the baseline algorithm, the method of the present invention has the advantages of fast convergence speed, high user experience quality convergence value, and more stability.

[0097] Reference Figure 5 According to the graph neural network module newly added by the method of the present invention, by comparing the user experience quality index of the MAPPO algorithm, it can be found that compared with the MAPPO algorithm, the method of the present invention has the advantages of fast convergence speed, high user experience quality convergence value, and greater stability.

[0098] Reference Figure 6 According to the transmission delay consumption of the method of the present invention, compared with the transmission delay consumption of the MAPPO algorithm, it can be found that compared with the MAPPO algorithm, the method of the present invention has the advantages of fast convergence speed, smaller transmission delay consumption convergence value, and more stability.

[0099] Reference Figure 7 ,According to the recommendation index of the method of the present invention, compared with the recommendation index of the collaborative filtering algorithm, it can be found that compared with the collaborative filtering algorithm, the method of the present invention has the advantages of fast convergence speed, smaller recommendation index and more stability.

[0100] Reference Figure 8 According to the number of intelligent agents in the method of the present invention, the user experience quality index of the baseline method (random algorithm, greedy algorithm, DQN algorithm, MADDPG algorithm) is compared. It can be found that compared with the baseline algorithm, as the number of intelligent agents increases, the method of the present invention has the advantages of fast convergence speed, high user experience quality convergence value, and more stability.

[0101] Reference Fig. 9 According to the number of videos of the method of the present invention, the user experience quality index of the baseline method (random algorithm, greedy algorithm, DQN algorithm, MADDPG algorithm) is compared. It can be found that compared with the baseline algorithm, as the number of videos increases, the method of the present invention has the advantages of fast convergence speed, high user experience quality convergence value, and more stability.

[0102] Reference Fig.10According to the user experience quality of the method of the present invention, compared with the algorithm without using the conflict resolution mechanism, it can be found that compared with the algorithm without using the conflict resolution mechanism, the method of the present invention has the advantages of fast convergence speed, higher convergence value of user experience quality value, and more stability.

[0103] Reference Fig.11 According to the user experience quality of the method of the present invention, compared with the algorithm without using the conflict resolution mechanism, it can be found that compared with the algorithm without using the conflict resolution mechanism, as the bandwidth from MBS to the roadside unit changes, the method of the present invention has the advantages of fast convergence speed, higher user experience quality value, and more stability.

[0104] Reference Figure 2 The present invention provides a system based on online joint recommendation and edge-assisted caching, comprising:

[0105] The original state setting module sets the original state characteristics of the agent based on task requirements;

[0106] A graph embedding module, which embeds the original state features of the agent based on a graph convolutional network to obtain a graph embedding of the agent;

[0107] A reinforcement learning module for inputting the agent's graph embedding into an actor-critic network to obtain the agent's recommended actions, cached actions, and rewards;

[0108] The decision output module takes the recommended action and cached action corresponding to the maximum reward as the decision output based on the reward maximization principle.

[0109] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0110] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A method based on online joint recommendation and edge-assisted caching, characterized in that: The following steps are involved: Set the original state characteristics of the agent based on task requirements; Embedding the original state features of the agent based on a graph convolutional network to obtain a graph embedding of the agent; Input the agent’s graph embedding into an actor-critic network to obtain the agent’s recommended actions, cached actions, and rewards; Based on the reward maximization principle, the recommended action and cache action corresponding to the maximum reward are used as decision outputs; The reward of the agent is expressed as follows: Among them, φ(t) represents the reward of the agent and also represents the quality experience of mobile phone users; f t (v j ,k) represents the number of mobile phone users v j and video content k as the recommendation score of the input; Indicates the video content k in the mobile phone user v at time period t j whether it is recommended; N represents the total set of roadside units; M represents the total set of mobile phone users; K represents the total set of video content k; Indicates the transmission of video content k to mobile phone user v j Total delay of Represents mobile phone users v j The nearest roadside unit at time t; Represents video content k from the nearest roadside unit Transmit to mobile phone usersv j transmission delay; represents the video content k transmitted from the main base station to the nearest roadside unit Delays; represents the number of mobile phone users v in time period t j The bitrate of cached content k; Indicates the nearest roadside unit at time period t-1 The bitrate of cached content k; α represents the coefficient that captures the relative importance of the recommendation score and transmission delay; Indicates whether transmission delay will occur; k Indicates the size of each video content; represents the delay of video content k being obtained from a neighboring roadside unit or from the main base station; i′ represents the number of the neighboring roadside unit; r i′ represents a neighbor roadside unit; Represents the roadside unit r i′ Whether the required video content k is cached; Indicates whether to obtain video content k from neighboring roadside unit i′ to mobile phone user v j ; Indicates whether to obtain video content k from the main base station to the mobile phone user v j ; B represents the main base station; Indicates that video content k is transmitted from roadside unit i′ to the nearest roadside unit transmission delay; represents the video content k transmitted from the main base station to the nearest roadside unit transmission delay.

2. The method based on online joint recommendation and edge-assisted caching according to claim 1, characterized in that: The original state features of the agent include video cache information, video recommendation information and coordinate information.

3. The method based on online joint recommendation and edge-assisted caching according to claim 1, characterized in that: The step of embedding the original state features of the agent based on the graph convolutional network to obtain the graph embedding of the agent specifically includes: Based on the graph convolutional network, the environment is designed as a graph, and the intelligent agent is used as a node on the graph; The nodes are connected based on the coverage of the agent, and the original state features of the agent are transmitted to the connected nodes along the connection line to obtain the propagation model of the graph convolutional network; The interconnected original state features are used as the input features of the agent, and the input features of the agent are subjected to several layers of graph message passing based on the propagation model of the graph convolutional network to obtain the graph embedding of the agent.

4. The method based on online joint recommendation and edge-assisted caching according to claim 1, characterized in that: The recommended actions and cached actions of the agent are expressed as follows: Among them, a n (t) represents the action space of the agent; Represents the roadside unit r at time period t i The bitrate of cached content k; represents the number of mobile phone users v in time period t j The bitrate of cached content k; Indicates the video content k in the time period for mobile phone users v j Whether it is recommended; K represents the total set of video content k.

5. The method based on online joint recommendation and edge-assisted caching according to claim 1, characterized in that: The transmission delay of the video content is affected by the transmission rate of the channel. The expression of the transmission rate of the channel is as follows: R Back (t)=min{R MBS (t),R CN (t)} in, represents the number of mobile phone users v in time period t j and roadside units i The transmission rate between Represents the roadside unit r i The channel bandwidth allocated to mobile phone users; Represents the roadside unit r i The signal transmission power of 2 represents the noise power; Indicates the time period t from the roadside unit r i To mobile phone users j The channel gain; ∈ represents the path loss coefficient; Represents the roadside unit r i The antenna gain factor; Represents mobile phone users v j With roadside unit i The distance between them; h0(t) represents the Rayleigh channel fading random variable that obeys the complex Gaussian distribution, and its value range is between 0 and 1; MBS represents the main base station; Represents the roadside unit r in time period t i and the transmission rate between the main base station; B0 represents the channel bandwidth allocated to the roadside unit by the main base station; P bs Indicates the signal transmission power of the primary base station; Represents the distance from the main base station to the roadside unit r in time period t i The channel gain of G bs Indicates the antenna gain coefficient of the primary base station; Represents the main base station and roadside unit r i The distance between CN (t) represents the transmission delay from the core network to the primary base station; C MBS Indicates the backhaul bandwidth when the number of roadside units connected to the main base station is less than or equal to the number of roadside units connected to the main base station; C CN represents the backhaul bandwidth when the number of roadside units connected to the main base station is greater than the number of roadside units connected to the main base station; N(t) represents the number of roadside units connected to the main base station in time period t; N Max Indicates the maximum number of roadside units carried by the main base station; R Back (t) represents the backhaul bandwidth from the core network to the roadside unit; R MBS (t) represents the backhaul bandwidth from the main base station to the roadside unit; R CN (t) represents the backhaul bandwidth from the core network to the master base station.

6. The method based on online joint recommendation and edge-assisted caching according to claim 1, characterized in that: When the number of roadside units without cached video content is higher than a set critical threshold, the roadside units and main base stations with video content with the smallest delay estimated from historical records are selected for video content recommendation and caching.

7. A system based on online joint recommendation and edge-assisted caching, characterized in that: The method for executing the method based on online joint recommendation and edge-assisted caching as claimed in claim 1 comprises: The original state setting module sets the original state characteristics of the agent based on task requirements; A graph embedding module, which embeds the original state features of the agent based on a graph convolutional network to obtain a graph embedding of the agent; A reinforcement learning module for inputting the agent's graph embedding into an actor-critic network to obtain the agent's recommended actions, cached actions, and rewards; The decision output module outputs the recommended action and cached action corresponding to the maximum reward as the decision output based on the reward maximization principle.

Images