A railway BIM data edge caching method based on multi-agent reinforcement learning

Abstract

The present invention provides a railway BIM data edge caching method based on multi-agent reinforcement learning. The method includes: constructing a BIMDipFM model, using the historical interaction information between users and BIM data under the BS to predict the retention rate, obtaining the content importance of BIM data, using the Markov game model to establish a multi-node collaborative caching model, obtaining the BIM data caching state and demand state of each cache node, generating the current moment environment state of the agent in the system, constructing an action mask module, filtering invalid actions, outputting an action space, outputting a determined action based on the actor network through the Gumble Softmax algorithm, the Agent performs the determined action in the environment, outputs a reward, obtains a new environment state, stores the environment state, determined action, reward and new environment state in an experience replay pool, performs experience replay after the experience replay pool is full, and updates the target network to improve the cache efficiency of BIM data, reduce the traffic load of the cloud side and the core network, and improve the user access speed.

Description

Technical Field

[0001] The present invention relates to the field of computer application technology, and in particular to a railway BIM data edge caching method based on multi-agent reinforcement learning. Background Art

[0002] Edge caching: is a technology that caches data and content at the edge of the network (usually an edge server). It stores frequently used data and content close to users to improve access speed and reduce network traffic.

[0003] Railway BIM (Building Information Modeling) applications are gradually evolving from the "worksite level" and "line level" stages to the "network level". Railway projects are linearly distributed, and the participants are dispersed, so the data information generated has the characteristics of multiple information sources and dispersed storage. At present, railway BIM data is characterized by large volume, and is currently mainly stored in a centralized manner on the cloud.

[0004] In terms of large-scale BIM data storage architecture, cloud computing, edge computing, distributed computing and other technologies are used to process and store BIM data in view of the large volume of BIM data. Some solutions have proposed a cloud-based BIM server framework that can dynamically merge and split BIM models. There are also solutions that have developed a cloud-based BIM data integration and management platform to support all participants to establish, store, manage and apply BIM data in a distributed manner according to their own data needs, so as to achieve data transmission and sharing. There are also solutions that have proposed a cloud-based BIMCloud framework to support collaboration and information sharing among all project participants. There are also solutions that use graph databases and graph theory to analyze the IFC (Industry Foundation Classes, an open standard for building information model data exchange) model and automatically convert the IFC model into a workflow of the attribute graph database. There are also related studies that have proposed using the cloud-end collaboration model to solve the transmission delay caused by using only cloud servers, such as cloud-end collaborative systems. However, when there is a large amount of real-time data transmission and processing, the synergy between the cloud and the end is not ideal. There are also plans to use the advantages of big data technology to manage BIM structured and unstructured data, establish a building information model storage system based on Hadoop and HBase big data environment, and implement basic management operations on IFC data.

[0005] In terms of BIM data scheduling, there are also plans to study the BIM large-scale multi-granularity interest scheduling algorithm, and develop a WebBIM multi-granularity online visualization integration platform and test platform for cloud-edge-page collaboration. The application model of "three terminals and one cloud" is studied, and a BIM construction management platform for information sharing among participants is built.

[0006] Deploying edge servers with caching functions at the edge of the network can greatly improve user access speed and reduce the traffic load of the core network. However, the edge server has limited storage capacity and it is impossible to cache all content. Therefore, how to design a caching strategy that meets the edge caching scenario is the key. Traditional caching strategies are based on static rules and do not consider the properties of the current content and the system status. Especially in the edge caching scenario, if the BIM content changes or the users within the coverage range of the communication node change, the user request pattern will change, resulting in a decrease in the cache hit rate and a reduction in the quality of user experience.

[0007] Traditional cache strategies, such as First In First Out (FIFO), are based on cache order, Least Recently Used (LRU), are based on access time, and Least Frequently Used (LFU), are based on usage frequency. These traditional cache algorithms are simple to implement and have low algorithm complexity. They have been widely used in network caches, but they ignore user behavior and request patterns. With the emergence of mobile edge computing technology and the continuous improvement of computing power, edge network nodes have certain computing and storage capabilities, making it possible to implement specific tasks such as data analysis and deep learning at the edge of mobile networks to understand user behavior and network characteristics. In recent years, more and more researchers have studied edge cache technology and proposed new cache solutions for different scenarios.

[0008] At present, the existing railway BIM data edge caching solutions include:

[0009] (1) Caching solution based on traditional content popularity model

[0010] Content value is an important factor to be considered in caching strategies, and the main criterion for evaluating content value is content popularity. Therefore, in recent years, content popularity models have been constructed based on network characteristics and content characteristics, and many new caching algorithms have been proposed. One scheme proposes an architecture for data analysis of content distribution networks, which uses video content features and the K-means algorithm to classify videos into five different popularity categories. The architecture uses dynamic model training and prediction services to provide relevant popular videos for caching for each content distribution network based on the latest trained model. Another scheme proposes a method for optimizing the state transition probability matrix of the Markov chain, and performs dynamic probability caching based on time-varying content popularity. This method improves the cache hit rate under changes in instantaneous content popularity. Another scheme designs a cooperative caching strategy for multi-unit networks, uses the spatiotemporal correlation of content popularity, establishes a dynamic probability model of content popularity, and proposes an online variational Bayes method for processing streaming data where content requests arrive in sequence. Another scheme proposes a caching scheme based on dynamic partitioning and content popularity. This scheme divides the nodes in the fog network into non-overlapping partitions to improve the content distribution in the network, and uses a larger popularity window to find popular content requests when making content caching decisions. In addition, the distance of the content provider is considered to place popular content near the end-user device. Another scheme proposes a content popularity and size-aware caching scheme in a hybrid heterogeneous network and analyzes the performance of the caching scheme in a non-uniform file size framework. The caching scheme can maximize the network average cache hit probability and the network average success probability affected by the available cache capacity.

[0011] (2) Caching solution based on deep learning

[0012] Using deep neural networks can better learn content features and user features, so many deep learning-based caching schemes have been proposed in recent years. Another scheme proposes an evolving caching algorithm based on deep learning and popularity awareness, which can adaptively learn the popularity of content that changes over time and determine which content should be replaced when the cache is full. The algorithm weights a large number of content features and trains multi-layer recurrent neural networks from shallow to deep when more requests arrive, reducing computational complexity. Another scheme proposes an active caching scheme based on point-to-point federated learning to protect the privacy of vehicles, and designs a double-weighted aggregation scheme to obtain higher global model accuracy. This scheme uses a variational autoencoder model based on collaborative filtering to predict future content popularity and pre-fetches predicted popular content in vehicles and roadside units to improve caching performance. Another scheme proposes an end-to-end deep learning framework for active caching and uses the interactions between users and content items and their features for learning. The framework can perform caching tasks by generating a probability distribution of each user's interaction with content items through a deep neural network, and supports centralized and distributed caching schemes. Another scheme proposes a learning-based hierarchical edge caching scheme, which captures the correlation and periodicity of request patterns between different contents in the time domain by designing a stacked autoencoder-long short-term memory network to improve the prediction accuracy of content popularity.

[0013] (3) Caching solution based on deep reinforcement learning

[0014] Many studies have used deep reinforcement learning-based methods to develop caching schemes and maximize system caching returns based on content popularity. The scenario where the popularity distribution is unknown and time-varying is studied, and a context-aware popularity learning algorithm is proposed, which can improve the speed and accuracy of obtaining content popularity. Then a caching scheme based on reinforcement learning is designed based on this algorithm. Another scheme considers online and offline factors, establishes a resource sharing model between devices, and then defines the optimization problem of maximizing the content offloading size and proposes an edge caching strategy based on a two-layer Q neural network. Another scheme studies the problem of collaborative edge caching in fog wireless access networks and proposes a federated deep reinforcement learning framework to learn content caching strategies. The framework learns the local caching model through a competitive Q network in each fog access point and finds the optimal caching strategy in a distributed manner. Another scheme proposes a weighted distributed reinforcement model to solve the cache replacement problem. The model can share the learned prediction model and enhance the adaptability of the model in a dynamic environment. Another scheme aims to minimize content access delay and traffic cost, and proposes an edge caching scheme based on multi-agent reinforcement learning. In this scheme, each edge node can adaptively learn its own optimal strategy and cache it together with other edge nodes.

[0015] At present, the disadvantages of the railway BIM data edge caching strategy in the above-mentioned prior art include:

[0016] In a multi-node edge cache scenario, the cache status of each node needs to be considered to make cache decisions, but edge nodes cannot obtain the global cache status. If a centralized processing method is used to collect the global cache status and train the cache decision model in the cloud, then each cache decision of the edge node needs to upload the local cache status to the cloud, and the cache decision model in the cloud will be used to determine the content to be updated and replaced by the edge node, which will increase communication overhead and reduce cache efficiency.

[0017] The existing edge caching strategy based on deep reinforcement learning models the caching decision problem as a Markov decision process, but most algorithms do not constrain the action space, which increases the difficulty for the caching decision network to select the optimal action.

[0018] The current strategy for replacing the content cached by edge servers is based on content importance prediction, which defines content importance based on the likelihood of a user requesting the content or the number of times the content is requested. User requests are affected by user roles (designers, construction personnel, operation and maintenance personnel) and needs. Users will not fully view the content of all data. If all nodes need to place complete data file content, this will result in a waste of pre-caching resources. Summary of the invention

[0019] An embodiment of the present invention provides a railway BIM data edge caching method based on multi-agent reinforcement learning, so as to effectively improve the caching efficiency of railway BIM data.

[0020] In order to achieve the above object, the present invention adopts the following technical scheme.

[0021] A railway BIM data edge caching method based on multi-agent reinforcement learning, comprising:

[0022] Step S10: construct a BIMDipFM model, use the historical interaction information between users and content under the edge server node BS to predict the retention rate, and obtain the content importance of the high-speed railway building information model BIM data;

[0023] Step S20: According to the importance of the content of the BIM data, a multi-node collaborative caching model is established using a Markov game model in a high-speed rail BIM multi-edge node caching scenario to obtain a caching state and a demand state of the BIM data of each caching node;

[0024] Step S30: Generate the current environmental state of the agent in the system according to the cache state and demand state of the BIM data of each cache node;

[0025] Step S40: construct an action mask module according to the current environmental state of the agent, filter invalid actions, reduce the action space, and output the action space;

[0026] Step S50, determining an action according to the action space based on the actor network through the Gumble Softmax algorithm output;

[0027] Step S60: the edge server Agent performs the determined action in the environment, the environment changes, calculates the reward according to the reward calculation formula, and outputs the reward;

[0028] Step S70, repeatedly executing the processing process of step S40 to step S60, obtaining a new environment state, and storing the environment state, the determined action, the reward and the new environment state as experience in an experience replay pool;

[0029] Step S80: After the experience replay pool is full, experience replay is performed to update the actor network and the critic network, and finally the target network is updated.

[0030] Preferably, the step S10 includes:

[0031] Obtain n high-speed rail BIM training data predicted by content retention rate, each data is recorded as (x, y), where x = {x1, x2, ..., x m}, y∈{0,1}, x is composed of m user-content features, which include content label discrete features and content maximum duration features. y indicates whether the user is retained. If the user browses the content for more than the set threshold or the maximum duration of the content, y=1, indicating that the user is retained, otherwise y=0, indicating that the user is not retained;

[0032] The BIMDipFM model includes an embedding representation module, an FM module, and a DNN module. Each piece of data in the historical interaction information contains a user ID and a content item ID. The user and content feature data are concatenated according to the ID to obtain the input feature. BIMDipFM uses the output of the Deep layer and the FM layer as the input of the sigmoid function to obtain the prediction result, which is expressed as shown in formula (1-1):

[0033] y=sigmoid(y FM +y DNN ) (1-1)

[0034] Among them, the output y value is the retention rate of each user for each content. According to the prediction results, the retention rate of a certain content f is obtained.i The average retention rate is shown in formula (1-2):

[0035]

[0036] The content importance is defined according to the average retention rate as shown in formula (1-3):

[0037] H={h1,h2,…,h M} (1-3)

[0038] Among them, h i Indicates the content i The importance of i is the content number.

[0039] Preferably, the step S20 includes:

[0040] According to the importance of the content of the BIM data, a multi-node collaborative caching model is established using a Markov game model in the high-speed rail BIM multi-edge node caching scenario to obtain the caching status and demand status of the BIM data of each caching node;

[0041] In the high-speed rail BIM multi-edge node caching scenario, the edge server of each BS node is an intelligent agent. The Markov decision process is extended to the multi-agent system, and the collaborative caching between multiple nodes is modeled as a Markov game model. N , P,γ,R1,…,R N ) is used to formally describe MG, N is the number of agents in the system; S is the system state, which refers to the joint state of multiple agents; A i , i∈{1,…,N} is the action space of the ith agent, and A is the joint action space of all agents, defined as A=A1×…×A N ; P is the state transfer function, which represents the probability distribution of entering the next state given the current system state and joint action; R i , i∈{1,…,N} is the reward function of the i-th agent, and the return and the state of the system at the next moment only depend on the current state and the joint action of multiple agents.

[0042] Preferably, the step S30 includes:

[0043] Each agent can only obtain its own cache status and user requests. The definition of the observation status of agent i at time t is shown in formula (1-4):

[0044]

[0045] in, j is the content f jThe serial number represents the cache status of the content under agent i; represents the user request sequence under agent i, a total of K requests; Indicates the importance of the content requested by agent i; is the size of the content requested by agent i. The size of the same content is the same for all agents. The global state at time t is shown in formula (1-5):

[0046]

[0047] Preferably, the step S40 includes:

[0048] The action space of each agent is the same, and the actions that agent i can choose at time t are The number indicates the proportion of cache content that needs to be updated. The calculation formula of the number is shown in formula (1-6):

[0049] q t =a t *0.1*len(L t ) (1-6)

[0050] Among them, len(L t ) represents the number of different contents in the request sequence, and the updated content is recorded as Indicates the first q after sorting from most to least important content t The different contents of the request sequence; the replacement content is recorded as The latter q indicates that the content in the BS node is of low importance t cached content;

[0051] By adding an action mask module after the output of the Q network to filter invalid actions, the AM module is based on the input state O at time t. t and user request sequence L t Get a valid action row vector, denoted as Aval = [a1, a2, ..., a n ], a i ∈{1, -∞}, where a i Indicates whether the i-th action in the action space is valid. If valid, a i =1, otherwise a i = -∞, by converting the Q value vector [q1, q2, ..., q n ] is multiplied by the corresponding element of Aval, setting the invalid action value to -∞, making the probability of the invalid action being selected 0.

[0052] Preferably, the step S50 includes:

[0053] Gumble Softmax reparameterizes discrete variables, removes sampling from the calculation of gradients without changing the original probability distribution, and adds constructed random variables to make the output discrete actions differentiable. The calculation expression of the Gumble Softmax algorithm is shown in formula (1-7):

[0054]

[0055] in is the output action μ i The probability distribution after softmax, G j = -log(-log(∈)), ∈~U(0,1) is a random variable that conforms to the Gumbel(0,1) distribution, j represents the discrete action in the action space, k=10 is defined, and there are 11 discrete actions in total, x i One-hot form representing the maximum probability action;

[0056] argmax operation x i It is not differentiable, and the Softmax algorithm is used to construct x i The continuous approximate variable of , the expression of the Softmax algorithm is shown in formula (1-8):

[0057]

[0058] Where τ is a temperature coefficient. The higher the temperature, the smoother the generated distribution. The lower the temperature, the closer the generated distribution is to the one-hot distribution. It can be derived by formula (1-9) to get the one-hot output. Add the computational graph;

[0059]

[0060] Among them, detach is able to x i Operations separated from the computation graph are then added to the computation graph Then you can pass Perform derivative calculations;

[0061] The Actor network is composed of a feedforward neural network. The output of the Actor network is reparameterized by Gumble Softmax. The Actor strategy network of agent i is denoted by μ i (a i |g i θ i ), according to the state g of the agent itself i Make a caching decision i .

[0062] Preferably, the step S60 includes:

[0063] The Critic network calculates a Q value as feedback to the Actor network to help the training and updating of the Critic network. The Critic network is the value network of the agent. The input of the Critic network includes the state and actions of the agent i itself, as well as the state and actions of the neighboring node agents, which are recorded as where s = [g1, g2, ..., g N ] represents the observation vector in the adjacent communication domain of agent i, a=[a1,a2,…,a N ] represents the set of actions performed by nodes in the adjacent communication domain of agent i;

[0064] The reward function of each agent is the same. Agent i completes the action at time t. The reward after is as shown in formula (1-10):

[0065]

[0066] in represents the set of updated content that does not exist in the adjacent communication domain at time t, indicating that the updated content is unique in this area. represents the content set that agent i has in the cache at time t and needs to update at time t, indicating that the update request has been cached. represents the set of content that agent i is replaced at time t-1 and needs to be updated at time t, indicating that the request to update is removed, h o The importance of content o, n o is the number of contents o, η, α and β are adjustable parameters.

[0067] Preferably, the step S70 includes:

[0068] Repeat the processing of step S40 to step S60 to obtain a new environment state, and the environment state changes from s at time t to s′ at time t+1. Put the four-tuple (s, a, r, s′) in the interaction process into the experience replay pool, where a represents the determined action and r represents the reward.

[0069] Preferably, the step S80 includes:

[0070] Using the temporal difference algorithm to improve the critic network For training, the TD target of the temporal difference algorithm is shown in formula (1-11):

[0071]

[0072] Where a′ represents the set of actions obtained by the agent through the target Actor network at the next moment, but not executed;

[0073] The TD error function is shown in formula (1-12), and the parameter w is updated using gradient descent i :

[0074]

[0075] Update the Actor network as shown in formula (1-13) according to the deterministic policy gradient ascent:

[0076]

[0077] After each round of current network update, the target network is updated using the soft update method as shown in formula (1-14):

[0078]

[0079] It can be seen from the technical solutions provided by the above-mentioned embodiments of the present invention that the present invention proposes an edge cache algorithm with centralized training and decentralized execution. After the algorithm is trained, each node can make cache decisions according to its own state through the local cache decision network, which reduces communication overhead and improves cache efficiency. The method of the present invention adds an action mask module after the cache decision network, which can filter invalid actions according to the current environmental state and request information, reduce the action space, and improve the performance and applicability of the cache model. The present invention uses the user retention rate of the content as the content importance to reflect the user's demand for the content, and establishes a content importance prediction model.

[0080] Additional aspects and advantages of the present invention will be given in part in the following description, which will become obvious from the following description, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0082] Figure 1 A schematic diagram of a multi-edge node caching scenario provided by an embodiment of the present invention;

[0083] Figure 2A schematic diagram of a decentralized edge caching algorithm MADGS (Multi-Agent Deep deterministic policy gradient with Gumble Softmax) provided in an embodiment of the present invention;

[0084] Figure 3 A schematic diagram of an action mask execution process provided by an embodiment of the present invention;

[0085] Figure 4 A schematic diagram of a BIMDipFM (BIM Data Importance Prediction Factorization Machines) model provided in an embodiment of the present invention;

[0086] Figure 5 A schematic diagram of an embedded representation module structure provided by an embodiment of the present invention.

[0087] Figure 6 A schematic diagram of a cache strategy framework provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0088] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0089] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or coupling. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.

[0090] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0091] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.

[0092] An embodiment of the present invention provides a multi-edge node caching scenario such as Figure 1 The network structure of the whole system is divided into three levels. The cloud server stores all the contents of the whole system, a total of M, denoted as F = {f1, f2, ..., f M}, considering the different sizes of each content, the content size is denoted as V = {v1, v2, ..., v M}, the cloud server communicates with the edge server node (hereinafter referred to as Base Station, BS) through the backhaul link.

[0093] There are N BS nodes at the edge, denoted as B = {b1, b2, ..., b N}, b i is the node with edge number i. Assume that the edge server cache size of each node is equal, denoted as C. Several BS nodes with similar geographical locations constitute an adjacent communication domain. Each BS node in the adjacent communication domain can communicate with each other to perform collaborative caching. i The users covered within the node network range are recorded as in Yes i The kth user in the node network range. Assume that in each time period, there are K users under each BS node making requests. For example, in time period t, b i The user request under the node is recorded as in is the request of the jth user. Since different users have different preferences and needs for content under different BS nodes, it is assumed that the importance of the content of each BS node is also different, denoted as H = {H1, H2, ..., H N}, where H i ={h i1 ,h i2 ,…,h iM} is b i The importance of the content under the node.

[0094] The cache state of the BS node is denoted as O = {O1, O2, ..., O N}, where O i ={o i1 , o i2 ,…,o iM},o ij ∈{0,1} is b i The cache status of the node, o ij =1 means in b i Content under node f j Already cached, o ij = 0 means not cached. The content cache status of the adjacent communication domain is denoted as D = {d1, d2, ..., d M}, where d j =1 indicates that f in adjacent communication domain j Already cached, d j = 0 means not cached. Assume that the entire system operation process can be divided into several time periods, recorded as

[0095] A processing flow chart of a railway BIM data edge caching method based on multi-agent reinforcement learning provided by an embodiment of the present invention is as follows: Figure 1 As shown, the process includes the following steps:

[0096] Step S10: First, build a BIMDipFM model, use the historical interaction information between users and content under the BS node to predict the retention rate, and build the content importance of high-speed rail BIM data.

[0097] Step S20: Then, according to the high-speed rail BIM multi-edge node caching scenario, a multi-node collaborative caching model is established using the Markov game model.

[0098] Step S30: Generate the current environmental state of the agent in the system according to the BIM data cache state and demand state of each cache node.

[0099] Step S40: According to the current environment state generated in step S30, an action mask module is constructed to filter invalid actions, reduce the action space, and output the action space.

[0100] Step S50: Based on the action space obtained in step S40, output a determined action through the actor network combined with Gumble Softmax technology. (Combining Gumble Softmax technology is innovative)

[0101] Step S60: The edge server Agent performs the determined action of step S50 in the environment, the environment changes, calculates the reward according to the reward calculation formula, and outputs the reward.

[0102] Step S70: reacquire the new environment state, and store the environment state, determined action, reward, and new environment state as experience in an experience replay pool.

[0103] Step S80: After the experience replay pool is full, experience replay is performed to update the actor network and the critic network. Finally, the target network is updated.

[0104] The execution process of the entire system is that when user u sends a request for content f, the local BS node where the user is located will first query whether the cache space of its server contains the content f. If it does, the content f will be directly returned to the user. If the local BS node does not find the content required by the user, it will forward the user request to other BS nodes in the adjacent communication domain. If other BS nodes find the content f, the other BS nodes will forward the content f to the user through the local BS node. If the content f is not found in other adjacent BS nodes, the user request will be forwarded to the cloud server, and the content f will be downloaded from the core network through the return link. According to the location where the content f is finally obtained, its delay T uf Calculate according to formula (4-1):

[0105]

[0106] Where T local <T adjoin <T cloud For users, the most desired thing is to find the required content on the local server, because this will reduce latency and provide a better user experience. The second option is to obtain the content from an adjacent server. If the content is obtained from a cloud server, not only will the latency be high, but the load on the backhaul link will also be increased.

[0107] The cache hit situation consists of two parts, one is the local server cache hit, and the other is the adjacent server cache hit. When calculating the cache hit rate, the number of adjacent server cache hits needs to be multiplied by the discount factor λ.

[0108] During the operation of the entire system, the edge caching strategy can be described by the set of caching decisions executed by each BS node in each time period, as shown in formula (4-2):

[0109] ∏={A1,A2,…,A N} (4-2)

[0110] Among them, A i Indicates b i Node in The cache decision within the time period is shown in formula (4-3):

[0111]

[0112] in, Yes i The cache update replacement action that the node needs to perform in the tth time period.

[0113] The present invention defines the cache problem with the goal of maximizing the overall cache hit rate in the future period of time. In the multi-edge node cache scenario, its definition is shown in formula (4-4):

[0114]

[0115]

[0116] in Indicates b i The cache status of the node, i represents the BS node with sequence number i, is the request of the jth user, in time period t, b i The user request under the node is recorded as F={f1,f2,…,f M}, indicating that the cloud server stores all the contents of the entire system, totaling M. Indicates the content f in the adjacent communication domain j Whether it is cached. λ represents the discount factor. K represents that in each time period, there are K users making requests under each BS node. N represents that there are N BS nodes at the edge. T represents a period of time. (4-4) represents maximizing the overall cache hit rate in the future. The st constraint is that in any time period, the cache capacity of each BS node must be less than the maximum cache C.

[0117] In the multi-edge node caching scenario, the edge server of each BS node is an agent, which extends the Markov decision process to a multi-agent system, and models the collaborative caching between multiple nodes as a Markov game model (MG). In the MG model, multiple agents interact with each other, and the decisions of these agents will affect each other's state and subsequent decisions. The MG model takes into account the interactions between agents and the uncertainties that may arise during the game. In this MG model, each agent can choose different actions, and these actions will affect the state transition of the system. At the same time, due to the existence of uncertainty, the reward function of each agent depends not only on its own actions, but also on the actions of other agents and the current system state. The goal of each agent is to maximize its long-term return, and the goal of the system is to balance the return of each agent to maximize the system's benefits.

[0118] In the MG model, we can usually use a tuple (N, S, A1, ..., A N , P,γ,R1,…,R N ) is used to formally describe MG. Where N is the number of agents in the system; S is the system state, generally referring to the joint state of multiple agents; A i , i∈{1,…,N} is the action space of the ith agent, and A is the joint action space of all agents, defined as A=A1×…×A N ; P is the state transfer function, which represents the probability distribution of entering the next state given the current system state and joint action; R i , i∈{1,…,N} is the reward function of the ith agent, and the reward function of each agent in the present invention is consistent. The reward and the state of the system at the next moment only depend on the current state and the joint action of multiple agents.

[0119] In multi-agent reinforcement learning, it is necessary to consider multi-node scenarios to define the state space, action space, and reward function.

[0120] State space: Each agent can only obtain its own cache state and user requests. The definition of the observed state of agent i at time t is shown in formula (4-5):

[0121]

[0122] in, j is the content f j The serial number represents the cache status of the content under agent i; represents the user request sequence under agent i, a total of K requests; Indicates the importance of the content requested by agent i; is the size of the content requested by agent i. The size of the same content is the same for all agents. The global state at time t is shown in formula (4-6):

[0123]

[0124] Action space: The action space of each agent is the same. The actions that agent i can choose at time t are Its value indicates the proportion of cache content that needs to be updated. The specific number is shown in formula (4-7):

[0125] q t =a t *0.1*len(L t ) (4-7)

[0126] Among them, len(L t ) represents the number of different contents in the request sequence, and the updated content is recorded as Indicates the first q after sorting from most to least important content t The different contents of the request sequence; the replacement content is recorded as The latter q indicates that the content in the BS node is of low importance t cached content.

[0127] Reward function: The reward function of each agent is the same. Agent i completes the action at time t. The reward after is shown in formula (4-8):

[0128]

[0129] in It represents the set of updated contents that do not exist in the adjacent communication domain at time t, indicating that the updated contents are unique in the area, which can greatly improve the system return. It represents the set of content that agent i has in the cache at time t and needs to update at time t, indicating that the update request has been cached. represents the set of content that agent i is replaced at time t-1 and needs to be updated at time t, indicating that the request to update is removed. o The importance of content o, n o is the number of contents o, η, α and β are adjustable parameters.

[0130] In order to reduce user latency as much as possible and make cache decisions on local BS nodes, and to use global state information in the game process of the Markov game model, the present invention proposes centralized training. A MADGS (Multi-Agent Deep deterministic policy gradient with Gumble Softmax, decentralized edge cache algorithm) provided in an embodiment of the present invention is as follows: Figure 2 As shown in the figure, the edge server of each BS node in the MADGS algorithm is an intelligent agent with an independent Actor network and Critic network. The Actor network is stored in the local edge server, and the Critic network is stored in the cloud server. The algorithm improves the multi-agent deep deterministic policy gradient algorithm so that its Actor network can output discrete actions. Centralized training means that each agent information is obtained through the Critic network of the cloud server for training. Decentralized execution means that after the training is completed, there is no need to communicate with the cloud server, and each agent can make cache decisions through its own Actor network.

[0131] Both the Actor network and the Critic network are composed of feedforward neural networks. Actot is the agent's policy network that can make cache decisions. Its output will be re-parameterized by Gumble Softmax (GS). The Actor policy network of agent i is denoted as μ i (a i |g i θ i ), can be based on the state of the agent itself g i Make a caching decision i The Critic network can calculate a Q value as feedback to the Actor network to help it train and update. The Critic network is only used during training and is the value network of the agent. Its input includes the state and actions of agent i itself, as well as the state and actions of the neighboring node agents, recorded as where s = [g1, g2, ..., g N ] represents the observation vector in the adjacent communication domain of agent i, a=[a1,a2,…,a N ] represents the set of actions performed by nodes in the adjacent communication domain of agent i.

[0132] Each agent’s Critic network and Actor network contain two forms: current network and target network. The target Actor network of the i-th agent is denoted as μ′ i (a′ i |g′ i θ′ i), used to sample the next state g′ based on the experience revisit pool i Get the next action a′ i , the target Critic network is recorded as Used to calculate the target Q value. The training data is obtained through the interaction between each agent and the environment. The interaction between the agent and the environment begins with K users making requests under each BS node at time t. After that, the agent will make cache action decisions based on the observed state. After all action decisions are completed, the action set a = [a1, a2, ..., a N ], and then feed the action a to be executed back to the environment, and obtain the reward r = [r1, r2, ..., r n ], the environment state changes from s at time t to s′ at time t+1. The four-tuple (s, a, r, s′) in the interaction process is put into the experience replay pool, and when a certain number is reached, the training of the Critic network and the Actor network begins.

[0133] Each round of training will train each agent in turn. Taking agent i as an example, first a certain amount of historical data is extracted from the experience replay pool, and the Critic network is trained using the Temporal Difference (TD) algorithm. For training, its TD target is shown in formula (4-9):

[0134]

[0135] Where a′ represents the set of actions obtained by the agent through the target Actor network at the next moment, but not executed.

[0136] The TD error function is shown in formula (4-10), and the parameter w is updated using gradient descent i :

[0137]

[0138] Then the Actot network is updated according to the deterministic policy gradient ascent as shown in formula (4-11):

[0139]

[0140] Finally, at the end of each round of current network update, the target network will be updated using the soft update method as shown in formula (4-12):

[0141]

[0142] The condition for the deterministic gradient to be updated is Existence, that is, action μ i (gi ) itself is i Differentiable, deterministic policy gradient algorithm outputs continuous actions. In order to output discrete actions, the usual practice is to convert the logical value of the action into the probability of each action through the softmax layer, and then use the argmax operation to output the action with the highest probability or sample the actions according to the probability. However, these two methods are not differentiable and cannot return gradients for updates.

[0143] The present invention reparameterizes discrete variables through Gumble Softmax, removes sampling from the calculation of gradients without changing the original probability distribution, and adds constructed random variables to make the output discrete actions differentiable. and variable G j A Gumbel-Max random variable can be constructed, and its expression is shown in formula (4-13):

[0144]

[0145] in is the output action μ i The probability distribution after softmax, G j = -log(-log(∈)), ∈~U(0,1) is a random variable that conforms to the Gumbel(0,1) distribution. j represents a discrete action in the action space. The present invention defines k=10 for a total of 11 discrete actions, x i Represents the one-hot form of the maximum probability action. Due to the argmax operation x i is not differentiable, the present invention constructs a Softmax function for x i The continuous approximate variable of is expressed as shown in formula (4-14):

[0146]

[0147] Where τ is a temperature coefficient. The higher the temperature, the smoother the generated distribution. The lower the temperature, the closer the generated distribution is to the one-hot distribution. It is differentiable, but it is still a continuous value. By using formula (4-15), we can get the one-hot output and convert it into Add to the computation graph.

[0148]

[0149] Among them, detach is able to x i Operations separated from the computation graph are then added to the computation graph Then you can use Perform derivative calculations.

[0150] The execution process of the MADGS edge caching algorithm is shown in Algorithm 1.

[0151]

[0152] In the Markov decision process of the above MADGS algorithm, both the Actor network and the Critic network are composed of a feedforward neural network Q network, and the optimal action value is obtained through the Q network. Its input is the current state s, and the output is the value of each action a. By adjusting the parameter w i To train the network. Since there may be invalid actions in the action space, for example, when the user request sequence is too long, actions larger than the cache capacity cannot be selected for execution. Therefore, the present invention proposes to filter invalid actions by adding an action mask module (AM) after the output of the Q network. An action mask execution process provided by an embodiment of the present invention is as follows Figure 3 shown.

[0153] The AM module can be based on the input state O at time t t and user request sequence L t Get a valid action row vector, denoted as Aval = [a1, a2, ..., a n ], a i ∈{1, -∞}, where a i Indicates whether the i-th action in the action space is valid. If valid, a i =1, otherwise a i = -∞. By converting the Q value vector [q1, q2, ..., q n ] can be multiplied by the corresponding element of Aval to set the invalid action value to -∞, making its probability of being selected 0.

[0154] The importance of content is affected by the frequency of use and interaction of users (designers, construction workers, operation and maintenance personnel, etc.). BIM data that is widely used is usually considered to have higher importance. Because BIM data is mainly three-dimensional geometric models and detailed attribute data, a station, a bridge, and a line integrate various professional data, so the volume is large, but users tend to only view part of the data related to their responsibilities, so the use of edge cache to store a small amount of frequently accessed data can effectively utilize storage resources, and is more conducive to reducing latency than centralized cloud storage. The present invention uses the average user retention rate of content as the importance of content. The higher the retention rate, the more cache resources are utilized. Whether the user is retained is divided according to the user's browsing time on the content he requested. There are n training data for content retention rate prediction, and each data is recorded as (x, y), where x = {x1, x2, ..., x m}, y∈{0,1}. x is composed of m user-content features, including discrete features such as content tags and continuous features such as the maximum duration of content. y indicates whether the user is retained. If the user browses the content for more than the set threshold or the maximum duration of the content, y=1 indicates that the user is retained, otherwise y=0 indicates that the user is not retained.

[0155] Build as Figure 4 The BIMDipFM (BIM Data Importance Prediction Factorization Machines) model shown in the figure predicts the retention rate by using the historical interaction information between users and content under the BS node. The BIMDipFM model mainly includes an embedding representation module, an FM module, and a DNN module. Each piece of data in the historical interaction information contains a user ID and a content item ID. The user and content feature data are concatenated according to the ID to obtain the input feature. The original input feature is a high-dimensional sparse feature, which is converted into a low-dimensional dense feature using the embedding representation module, and the converted feature is input into the FM module and the DNN module. The BIMDipFM module adds the outputs of the two as a whole and obtains the output through a sigmoid nonlinear mapping, that is, the retention rate of each user for the interactive content. The average retention rate is calculated as the content importance, denoted as H.

[0156] Among them, the embedding representation module is a fully connected neural network, such as Figure 5 As shown, each input feature x i The value range of is different, the input feature dimension d is also different, but the output dense feature e i The dimension k of is the same as k, for example k = 4. This module can be learned by gradient back propagation.

[0157] The feature expression of feature x after embedding the representation module is as shown in formula (4-16):

[0158] e=[e1,e2,…,e m ] (4-16)

[0159] Among them, e i It represents the value of the i-th feature in feature x after embedding, which is a k-dimensional vector.

[0160] In addition to the embedding representation module, the BIMDipFM network structure also includes two parts: the FM module and the DNN module, which can learn the low-order and high-order combination features of the data respectively. The parameters and output features of the embedding representation module are shared by the FM module and the Deep module.

[0161] The FM module is a model based on matrix decomposition. It captures the association between features by modeling the quadratic cross terms of features, thereby improving the generalization ability of the model. The FM module consists of an addition unit and several inner product units. The output after the module is shown in formula (4-17):

[0162]

[0163] Among them, w0 is the bias term, w i is the weight, w i e i The additive term reflects the relationship of the first-order characteristics; V i is the latent vector of the i-th dimension feature of e, V j is the latent vector of the j-th dimension feature of e, <V i , V j >The inner product unit reflects the influence of the second-order features.

[0164] The DNN module is a feedforward neural network used to extract high-order features. Its input data is the feature e processed by the embedding layer. The output after passing through this module is shown in formula (4-18):

[0165] y DNN =σ(W |H|+1 ·e H +b |H|+1 ) (4-18)

[0166] Where H is the hidden layer depth, e H , W, b are the output, weight, and bias of the hidden layer respectively.

[0167] Finally, BIMDipFM uses the output of the Deep layer and the FM layer as the input of the sigmoid function to obtain the prediction result, which is expressed as shown in formula (4-19):

[0168] y=sigmoid(y FM +y DNN ) (4-19)

[0169] The output y value is the retention rate of each user for each content. According to the prediction results, we can get the retention rate of a certain content f i The average retention rate is shown in formula (4-20):

[0170]

[0171] The content importance is defined according to the average retention rate as shown in formula (4-21):

[0172] H={h1,h2,…,h M} (4-21)

[0173] Among them, h i Indicates the content i The importance of i is the content number.

[0174] The present invention calculates the content retention rate and deploys it on the cloud side, and adds its value as the environmental state information to S as the content importance component H, as shown in formula (4-5). The overall cache strategy framework is as follows: Figure 6 shown.

[0175] In summary, the railway BIM data in the embodiment of the present invention has the problems of large volume, centralized storage, and difficulty in loading. The existing technology has established a BIM data integration and management platform based on cloud computing, and the data is stored on the cloud side and the BIM model of the terminal is loaded through technologies such as WebGL. When there is a large amount of real-time data transmission and processing, the synergy between the cloud and the terminal is not ideal. The present invention proposes a three-layer distributed storage architecture of cloud-edge-end. By deploying edge servers with caching functions at the edge of the network and storing some BIM data, it can greatly improve the user access speed and reduce the traffic load on the cloud side and the core network.

[0176] Traditional cache strategies, such as the first-in-first-out strategy, the least recently used strategy, and the least used strategy, are simple to implement and have low algorithm complexity. They have been widely used in single-node edge caches, but they are not suitable for global cache optimization of multiple edge nodes.

[0177] Existing multi-node edge caching methods, including deep learning and reinforcement learning methods, take into account the global cache state, but mostly use centralized processing to collect the global cache state and train the cache decision model in the cloud, which will increase communication overhead and reduce cache efficiency. The present invention proposes an edge caching algorithm with centralized training and decentralized execution. After the algorithm is trained, each node can make cache decisions based on its own state through the local cache decision network, reducing communication overhead and improving cache efficiency. In addition, in the efficiency optimization of the method, the present invention proposes to add an action mask module after the cache decision network, which can filter invalid actions according to the current environmental status and request information, reduce the action space, and improve the performance of the cache model.

[0178] Most of the existing scenarios that use edge nodes for caching are for streaming data such as videos. The present invention caches the loading and use of railway BIM geometric models. Therefore, the cached content characteristics are different from other scenarios. As the action decision of each edge node intelligent agent that needs to be considered in the edge caching strategy, the present invention proposes to use the user retention rate of the content as the content importance to reflect the user's demand for the content, and establishes a content importance prediction model.

[0179] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0180] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention or certain parts of the embodiments.

[0181] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0182] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A railway BIM data edge caching method based on multi-agent reinforcement learning, characterized in that: include: Step S10: construct a BIMDipFM model, use the historical interaction information between users and content under the edge server node BS to predict the retention rate, and obtain the content importance of the high-speed railway building information model BIM data; Step S20: According to the importance of the content of the BIM data, a multi-node collaborative caching model is established using a Markov game model in a high-speed rail BIM multi-edge node caching scenario to obtain a caching state and a demand state of the BIM data of each caching node; Step S30: Generate the current environmental state of the agent in the system according to the cache state and demand state of the BIM data of each cache node; Step S40: construct an action mask module according to the current environmental state of the agent, filter invalid actions, reduce the action space, and output the action space; Step S50, determining an action according to the action space based on the actor network through the Gumble Softmax algorithm output; Step S60: the edge server Agent performs the determined action in the environment, the environment changes, calculates the reward according to the reward calculation formula, and outputs the reward; Step S70, repeatedly executing the processing process of step S40 to step S60, obtaining a new environment state, and storing the environment state, the determined action, the reward and the new environment state as experience in an experience replay pool; Step S80: After the experience replay pool is full, experience replay is performed to update the actor network and the critic network, and finally the target network is updated; The step S10 comprises: Obtain n high-speed rail BIM training data predicted by content retention rate, each data is recorded as (x, y), where x = {x1, x2, …, x m }, y∈{0,1}, x is composed of m user-content features, which include content label discrete features and content maximum duration features. y indicates whether the user is retained. If the user browses the content for more than the set threshold or the maximum duration of the content, y=1, indicating that the user is retained, otherwise y=0, indicating that the user is not retained; The BIMDipFM model includes an embedding representation module, an FM module, and a DNN module. Each piece of data in the historical interaction information contains a user ID and a content item ID. The user and content feature data are concatenated according to the ID to obtain the input feature. BIMDipFM uses the output of the Deep layer and the FM layer as the input of the sigmoid function to obtain the prediction result, which is expressed as shown in formula (1-1): y=sigmoid(y FM +y DNN )#(1-1) Among them, the output y value is the retention rate of each user for each content. According to the prediction results, the retention rate of a certain content f is obtained. i The average retention rate is shown in formula (1-2): The content importance is defined according to the average retention rate as shown in formula (1-3): H={h1,h2,…,h M }#(1-3) Among them, h i Indicates the content i The importance of i is the content number.

2. The method according to claim 1, characterized in that The step S20 includes: According to the importance of the content of the BIM data, a multi-node collaborative caching model is established using a Markov game model in the high-speed rail BIM multi-edge node caching scenario to obtain the caching status and demand status of the BIM data of each caching node; In the high-speed rail BIM multi-edge node caching scenario, the edge server of each BS node is an intelligent agent. The Markov decision process is extended to a multi-agent system, and the collaborative caching between multiple nodes is modeled as a Markov game model. N ,P,γ,R1,…,R N ) is used to formally describe MG, N is the number of agents in the system; S is the system state, which refers to the joint state of multiple agents; A i , i∈{1,…,N} is the action space of the ith agent, and A is the joint action space of all agents, defined as A=A1×…×A N ; P is the state transfer function, which represents the probability distribution of entering the next state given the current system state and joint action; R i ,i∈{1,…,N} is the reward function of the i-th agent. The return and the state of the system at the next moment only depend on the current state and the joint action of multiple agents.

3. The method according to claim 2, characterized in that The step S30 includes: Each agent can only obtain its own cache status and user requests. The definition of the observation status of agent i at time t is shown in formula (1-4): in, j is the content f j The serial number represents the cache status of the content under agent i; represents the user request sequence under agent i, a total of K requests; Indicates the importance of the content requested by agent i; is the size of the content requested by agent i. The size of the same content is the same for all agents. The global state at time t is shown in formula (1-5):

4. The method according to claim 3, characterized in that The step S40 includes: The action space of each agent is the same, and the actions that agent i can choose at time t are The number indicates the proportion of cache content that needs to be updated. The calculation formula of the number is shown in formula (1-6): q t =a t *0.1*len(L t )#(1-6) Among them, len(L t ) represents the number of different contents in the request sequence, and the updated content is recorded as Indicates the first q after sorting from most to least important content t The different contents of the request sequence; the replacement content is recorded as The latter q indicates that the content in the BS node is of low importance t cached content; By adding an action mask module after the output of the Q network to filter invalid actions, the AM module is based on the input state O at time t. t and user request sequence L t Get a valid action row vector, denoted as Aval = [a1, a2, ..., a n ],a i ∈{1,―∞}, where a i Indicates whether the i-th action in the action space is valid. If valid, a i =1, otherwise a i = -∞, by Q network output Q value vector [q1,q2,…,q n ] is multiplied by the corresponding element of Aval, setting the invalid action value to -∞, making the probability of the invalid action being selected 0.

5. The method according to claim 4, characterized in that The step S50 includes: Gumble Softmax reparameterizes discrete variables, removes sampling from the calculation of gradients without changing the original probability distribution, and adds constructed random variables to make the output discrete actions differentiable. The calculation expression of the Gumble Softmax algorithm is shown in formula (1-7): in is the output action μ i The probability distribution after softmax, G j =―log(―log(∈)),∈~U(0,1) is a random variable that conforms to the Gumbel(0,1) distribution, j represents the discrete action in the action space, k=10 is defined, and there are 11 discrete actions in total, x i One-hot form representing the maximum probability action; argmax operation x i It is not differentiable, and the Softmax algorithm is used to construct x i The continuous approximate variable of , the expression of the Softmax algorithm is shown in formula (1-8): Where τ is a temperature coefficient. The higher the temperature, the smoother the generated distribution. The lower the temperature, the closer the generated distribution is to the one-hot distribution. It can be derived by formula (1-9) to get the one-hot output. Add the computational graph; Among them, detach is able to x i Operations separated from the computation graph are then added to the computation graph Then you can pass Perform derivative calculations; The Actor network is composed of a feedforward neural network. The output of the Actor network is reparameterized by Gumble Softmax. The Actor strategy network of agent i is denoted as μ i (a i ∣g i θ i ), according to the state g of the agent itself i Make a caching decision i .

6. The method according to claim 5, characterized in that The step S60 includes: The Critic network calculates a Q value as feedback to the Actor network to help the training and updating of the Critic network. The Critic network is the value network of the agent. The input of the Critic network includes the state and actions of the agent i itself, as well as the state and actions of the neighboring node agents, which are recorded as where s = [g1, g2, …, g N ] represents the observation vector in the adjacent communication domain of agent i, a=[a1,a2,…,a N ] represents the set of actions performed by nodes in the adjacent communication domain of agent i; The reward function of each agent is the same. Agent i completes the action at time t. The reward after is as shown in formula (1-10): in represents the set of updated content that does not exist in the adjacent communication domain at time t, indicating that the updated content is unique in this area. represents the content set that agent i has in the cache at time t and needs to update at time t, indicating that the update request has been cached. represents the set of content that agent i is replaced at time t-1 and needs to be updated at time t, indicating that the request to update is removed, h o To update the importance of a content o in a content collection, n o is the number of contents o, η, α and β are adjustable parameters.

7. The method according to claim 6, characterized in that The step S70 includes: Repeat the processing of step S40 to step S60 to obtain a new environment state, and the environment state changes from s at time t to s′ at time t+1. Put the four-tuple (s, a, r, s′) in the interaction process into the experience replay pool, where a represents the determined action and r represents the reward.

8. The method according to claim 7, characterized in that The step S80 includes: Using the temporal difference algorithm to improve the critic network For training, the TD target of the temporal difference algorithm is shown in formula (1-11): Where a′ represents the set of actions obtained by the agent through the target Actor network at the next moment, but not executed; The TD error function is shown in formula (1-12), and the parameter w is updated using gradient descent i : Update the Actor network as shown in formula (1-13) according to the deterministic policy gradient ascent: After each round of current network update, the target network is updated using the soft update method as shown in formula (1-14):

Images