Motor train unit video collaborative caching method based on multi-agent reinforcement learning

Abstract

The invention discloses a motor train unit video collaborative caching method based on multi-agent reinforcement learning. The method comprises the following steps: constructing a motor train unit multi-compartment collaborative caching video service system; defining a request response time cost according to a user request cache hit condition; defining a resource replacement time cost according to the difference of cached video contents in adjacent time periods; the problem of minimizing the video service time cost of the motor train unit is formalized; and calling a multi-carriage collaborative caching algorithm based on multi-agent reinforcement learning, and determining a video content caching strategy of each time period. According to the invention, the video content is cached on the motor train unit carriages, the difference of different carriages and the problem of linear networking are considered, the multi-carriage collaborative caching system and method are designed, the acquisition of the video content by the user is accelerated, and the utilization rate of the caching space is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video data caching, and in particular to a method for collaborative caching of EMU videos based on multi-agent reinforcement learning. Background Art

[0002] With the rapid development of high-speed rail networks, EMUs have become an important means of transportation for people. In recent years, videos have become popular due to their direct expression and easy consumption, and have become a major component of Internet traffic. People prefer to obtain information through videos in their daily lives. Therefore, video services in EMU carriages have become increasingly important. However, the surge in traffic brought by videos has put higher requirements on network bandwidth and latency, especially in mobile environments such as EMUs, where network connections may be more unstable and the uncertainty of user experience quality has a greater impact. Video services in EMU scenarios face a series of challenges such as unstable network communication environment, high passenger mobility, and high dynamic variability of resources.

[0003] By caching video content at the edge of the carriage, the viewing experience of passengers can be significantly improved, and buffering and freezing problems caused by network latency and bandwidth limitations can be reduced. By deploying cache servers at the edge of the carriage, video content can be preloaded closer to the passengers, thereby reducing the distance and time of data transmission and speeding up video loading.

[0004] However, previous studies often considered optimizing the EMU network communication environment in terms of channels or base stations to speed up the rate at which users obtain video content, but rarely considered how to provide users with more stable and reliable video services from the perspective of EMU edge cache. The main reasons for this phenomenon are as follows: First, the attributes of users in the EMU carriages vary greatly, which are often reflected in the user's seat class, identity, occupation, and cultural characteristics that vary with the region, resulting in large differences in the content preferences of users in different carriages. Secondly, the mobility of users in EMU carriages is greater than that of users around traditional servers, resulting in large dynamic variability of resources, and the size of dynamic variability varies between different carriages. This makes algorithms such as LRU and LFU, which essentially want to learn resource request patterns through training on historical data, difficult to use for the future when faced with EMU scenarios with large differences and dynamics, and the effect of prediction-based caching algorithms is average.

[0005] Second, the current EMU carriages do not have much design for the edge cache architecture, and the single-carriage service architecture design often has the problem of high load pressure and inability to dynamically adjust.

[0006] Based on the above reasons, although caching video content at the edge of the car can significantly improve the passengers' viewing experience, there are still many difficulties in realizing this technical means. Summary of the invention

[0007] The present invention aims to solve one of the technical problems existing in the related art at least to a certain extent.

[0008] One object of the present invention is to provide a method for collaborative caching of EMU videos based on multi-agent reinforcement learning, which takes into account the differences between different carriages and the problem of linear networking, and performs collaborative caching of videos among multiple carriages, making the carriage video service more accurate and fast, while balancing the load of each server and fully improving the utilization rate of the EMU cache space.

[0009] In order to achieve the above-mentioned object, the present invention provides a method for collaborative caching of EMU videos based on multi-agent reinforcement learning, comprising the following steps: S100. Constructing a multi-carriage collaborative caching video service system for EMUs, including defining a set of EMU carriages and a set of video content, and dividing the EMU running time into time periods according to the travel time between stations; S200. Define the request response time cost according to the user request cache hit situation; S300 defines the resource replacement time cost according to the difference in cached video content in adjacent time periods; S400. Formalize the time cost minimization problem of EMU video service; S500. Call a multi-carriage collaborative caching algorithm based on multi-agent reinforcement learning to determine the video content caching strategy for each time period, where each car is regarded as an agent, a neural network is used to fit the agent's actions, and two evaluation networks are set for collaborative optimization; the data of all agents in the time period are stored in the sample pool in the form of five-tuples, including the global state, global action, the global state of the next time period, the reward set and the end mark; if the number of samples reaches the minimum batch processing threshold, the neural network is trained and the target network parameters are updated.

[0010] A further preferred technical solution of the present invention is that the construction of the EMU multi-carriage collaborative caching video service system in step S100 includes: S110, define the EMU carriage set and video content set, and number the carriages from the front to the rear ,set up Represents a collection of EMU carriages, Represents a collection of video content; S120, divide the EMU running time according to stations, and each period Represents the travel time between the current station and the next station; S130, constructing EMU video service includes: S131. Set up a carriage cache server as a reverse proxy in each carriage, and define the upper limit of the cache capacity of each carriage cache server as , Each carriage cache server is connected to the external base station through the roof antenna of the carriage where it is located. To communicate; users connect to the train LAN through the wireless access points set up in each carriage; the train sets up a central server to query the deployment of the train video content; S132, when the carriage When a user in the train accesses the network and requests video content, the user requests a cache hit, and the user is sent to the user from the train, other trains or an external base station; S133, Order period Interior All users in the video content The number of requests is , define a binary indicator variable , indicating that Time slot car All users on the video content Whether the request is made by the target Response, if by the target Response, then ,otherwise, ,in , Cache server or external base station for EMU carriages.

[0011] As a preferred embodiment, when the carriage is When a user in the train accesses the network and requests video content, the user requests a cache hit, and the user's requested video content is sent to the user from the train, other trains or external base stations; the specific method is: S132-a, determining whether the video content requested by the user is cached in the car where the user is located, if so, executing step S132-e, otherwise executing step S132-b; S132-b, obtaining the deployment status of the video content requested by the user in the EMU from the central server, if the EMU has a cache of the video content requested by the user, executing step S132-c, otherwise executing step S132-d; S132-c, select the car that is closest to the car that issued the request and has cached the video content requested by the user to add to the candidate response car set, check whether there are two optional candidate cars with the same distance, if not, directly select the car that is closest to the car that issued the request and has cached the video content requested by the user as the response car, if so, select the car with the smaller load among the two candidate cars as the response car, and the car where the user is located acts as a proxy to request the video content from the response car, and execute step S132-e after receiving the response; S132-d, the carriage where the user is located acts as an agent to request the video content required by the user from the external base station, and after receiving the response, executes step S132-e; S132-e. The carriage where the user is located sends the video content requested by the user to the users in the same carriage.

[0012] Preferably, the step S200 defines the request response time cost according to the cache hit situation of the user request; specifically includes: According to the cache hit situation of the user request, there are three different responses, namely, in-car response, inter-car response and external base station response, corresponding to three different request response time costs; Define binary indicator variables , indicating the carriage and carriages Whether the request between passes through the carriage and carriages The connected link interval, if passing through ,otherwise ,Right now: (1); Assume the total link bandwidth is , define the compartment To the target The request response time cost is: (2); in, The response time cost of the car requesting video content from the external base station is set to a fixed value; Indicates time period Interior All users in the video content Number of requests; Indicated in Time slot car All users on the video content Whether the request is made by compartment Response, if by carriage Response, then ,otherwise, ; Indicates the carriage and carriages Whether the request between passes through the carriage and goals The link interval connected when it is a carriage; In the period Satisfaction Carriage The request response time cost for all users is: (3).

[0013] Preferably, the step S300 defines the resource replacement time cost according to the difference of cached video content in adjacent time periods; specifically includes: S310. Define binary indicator variables ,express Video content of the period Is it cached in the carriage? On, if The video content of the time period is cached in the carriage On, then ,otherwise, ; S320, in each period At the beginning, if the video content to be cached in the current period is the same as that in the previous period If the cached video content is different, each carriage cache server obtains the video content required for the current period from other carriage cache servers or external base stations to replace the previous period. Cached video content, defined in time periods Time Car The resource replacement time cost is: (4).

[0014] Preferably, the formalized EMU video service time cost minimization problem in step S400 is specifically: Let the EMU video service time cost be the weighted sum of the request response time cost and resource replacement time cost of all carriages in all time periods, and formalize the EMU video service time cost minimization problem as follows: (5); Constraints: (6); (7); (8); (9); (10); in, and is a weight factor used to adjust the weight between the request response time cost and the resource replacement time cost. Indicates the carriage The upper limit of the cache capacity of the cache server; Indicated in Time slot car All users on the video content Whether the request is made by compartment Response, if by carriage Response, then ,otherwise, .

[0015] Preferably, step S500 includes: S510, each carriage is regarded as an intelligent agent, and the carriage number is Represents the intelligent agent corresponding to the carriage; uses a neural network to fit the intelligent agent The action is recorded as the training action network ,in For intelligent agents The training action network parameters of the corresponding target action network are recorded as ; For each agent , set up two training and evaluation networks, respectively denoted as and ,in For intelligent agents The first training evaluation network parameters, the corresponding target network is ,and For intelligent agents The second training evaluation network parameters of the corresponding target network are ; S520, define each intelligent agent exist The status of the time period, including The resource cache status of the carriage itself during the time period, User requests, carriage attributes and The location attributes of the time period; The carriage attribute set is , the location attribute set is , then the agent exist The state of a time period is represented by ,in Representing an Agent The properties of the car you are in. Represents the agent Current location attributes, , indicating the carriage The cache server is The cache status of the time period, Indicates the carriage User in Requests for video content during a time period; S530, according to the obtained intelligent agent The state of the time period, training action network , output each video content The cache probability of each video content Sort the cache probabilities from large to small and select the first The video content is used as the final cache action; Define the action of the agent as , where if the agent exist Time period cache content ,but ,otherwise, ;and ; S540, each agent Execute an action , the status is Enter the next state ; The environment gives rewards , defined as Time-segment Agent The negative value of the weighted sum of the request response time cost and the resource replacement time cost is expressed as: (11); S550, place all agents in The data of the time period is removed by the time attribute. Stored in the sample pool in the form of five tuples ,in, , represents the global state composed of the local observation states of all agents; , represents the global action composed of the local actions of all agents; , indicating the global state of the next period; , represents the reward set; is a binary indicator variable. If the EMU arrives at the terminal station, ,otherwise, ; S560, if the number of samples reaches the minimum batch processing threshold , train the neural network and update the target network parameters.

[0016] Preferably, the training of the neural network in step S560 comprises the following specific steps: S561, first calculate the agent The temporal difference error of the two evaluation networks is: (12); (13); in, is the discount factor; For intelligent agents Rewards; Represents the relative Action for the next period; Huber error is selected as the final error ,in is the evaluation network number, expressed as: (14); S562, every time Random samples are batch processed and the training and evaluation network parameters are updated by gradient descent, which can be expressed as: (15); in, is the step length; S563, for the update of the strategy, gradient ascent is used; for the training action network and all target networks, the update is delayed, that is, each update Train the evaluation network again, and then update the training action network and all target networks, expressed as: (16); in, is the step length; For intelligent agents The local observation state.

[0017] S564, using a soft update method to update the parameters of the target network, expressed as: (17); (18); in, is a coefficient between 0 and 1 that controls the step size of the update; soft updates slowly follow the parameter changes of the training network by gradually adjusting the parameters of the target network.

[0018] Beneficial effect: The present invention can greatly reduce the time for users to obtain video content by caching video content on the EMU carriage.

[0019] The present invention takes into account the large differences in content preferences of users in different carriages in the EMU scenario, as well as the large differences in dynamic changes in requests from users in different carriages, and designs a multi-carriage collaborative caching system, which can make the car video service more accurate and fast, while balancing the load of each server and fully improving the utilization rate of the EMU cache space.

[0020] The present invention analyzes the linear networking of EMUs, making the defined cross-carriage hit response time cost more realistic. At the same time, the present invention considers the problem of poor communication conditions along the EMU, defines the resource replacement time cost, so that the resource deployment not only considers the current, but also takes into account possible future user request requirements, freeing up bandwidth for resources that really need to be downloaded and cached. Through the constraints and checks and balances between the request response time cost and the resource replacement time cost, the resource deployment is more reasonable.

[0021] Due to the large differences and dynamics of EMU scenarios, learning of historical rules is usually difficult to apply to the future, which makes the prediction-based caching algorithm generally ineffective. From the perspective of decision-making, the present invention adopts a multi-agent reinforcement learning-based technology to decide on the resource caching strategy, which can better cope with the dynamics and differences of environmental changes.

[0022] The present invention uses multi-agent reinforcement learning technology and also adds special indicators for the EMU scenario, making action decisions more accurate and efficient. A series of optimization methods are used to ensure the smoothness and accuracy of the training process and to deal with the overestimation problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is an overall flow chart of the EMU video collaborative caching method based on multi-agent reinforcement learning of the present invention.

[0024] Figure 2 It is a schematic diagram of the EMU multi-carriage collaborative caching video service system constructed by the present invention.

[0025] Figure 3 The present invention is a flowchart of user request response processing of a multiple-carriage collaborative caching video service system for a train set constructed by the present invention.

[0026] Figure 4 It is a flow chart of the multi-carriage collaborative caching algorithm based on multi-agent reinforcement learning of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments, and they should not be understood as limitations on the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In the description of the present invention, it should be understood that the terms used are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0028] Example: The following combination Figure 1-Figure 4 The EMU video collaborative caching method based on multi-agent reinforcement learning provided in this embodiment is specifically described.

[0029] The EMU video collaborative caching method based on multi-agent reinforcement learning in this embodiment is as follows: Figure 1 As shown, the main steps include: S100. Construct a multi-carriage collaborative caching video service system for EMUs; S200. Define the request response time cost according to the user request cache hit situation; S300 defines the resource replacement time cost according to the difference in cached video content in adjacent time periods; S400. Formalize the time cost minimization problem of EMU video service; S500. Call a multi-carriage collaborative caching algorithm based on multi-agent reinforcement learning to determine the video content caching strategy for each time period.

[0030] S100 builds a multi-carriage collaborative caching video service system for EMUs, such as Figure 2 As shown, including: S110, define a set of train carriages and a set of video content, and number the carriages from the front to the rear in sequence: ,set up Represents a collection of EMU carriages, Represents a collection of video content.

[0031] S120, divide the EMU running time according to stations, and each period Represents the travel time from the current stop to the next stop.

[0032] S130, constructing EMU video service includes: S131. Set up a carriage cache server as a reverse proxy in each carriage, and define the upper limit of the cache capacity of each carriage cache server as , ; It can be used to respond to the user's request for video content on the EMU; the cache servers in each carriage are connected by wires to form an EMU local area network, and each carriage cache server communicates with the external base station through the roof antenna of the carriage where it is located Communicate; users connect to the EMU LAN through the wireless access points set up in each carriage; the EMU sets up a central server to query the deployment of EMU video content and perform other services during the operation of the EMU.

[0033] S132, when the carriage When a user in the train accesses the network of this compartment and requests video content, the user requests a cache hit, and the user's requested video content is sent to the user from this compartment, other compartments or external base stations; Figure 3 As shown, including: S132-a. Determine whether the video content requested by the user is cached in the car where the user is located. If so, execute step S132-e; otherwise, execute step S132-b.

[0034] S132-b. Obtain the deployment status of the user-requested video content in the EMU from the central server. If the EMU has a cache of the user-requested video content, execute step S132-c; otherwise, execute step S132-d.

[0035] S132-c, select the car that is closest to the car that issued the request and has cached the video content requested by the user to add to the candidate response car set, check whether there are two optional candidate cars with the same distance, if not, directly select the car that is closest to the car that issued the request and has cached the video content requested by the user as the response car, if so, select the car with the smaller load among the two candidate cars as the response car, and the car where the user is located acts as a proxy to request the video content from the response car, and execute step S132-e after receiving the response;

[0036] S132-d. The car where the user is located acts as an agent to request the video content required by the user from the external base station, and executes step S132-e after receiving a response.

[0037] S132-e. The carriage where the user is located sends the video content requested by the user to the users in the same carriage.

[0038] S133, Order period Interior All users in the video content The number of requests is , define a binary indicator variable , indicating that Time slot car All users on the video content Whether the request is made by the target Response, if by the target Response, then ,otherwise, ,in , It can be a cache server in the EMU carriage or an external base station.

[0039] Step S200 defines the request response time cost according to the cache hit situation of the user request; specifically includes: According to the cache hit situation of the user request, there are three different responses, namely, in-car response, inter-car response and external base station response, corresponding to three different request response time costs; Define binary indicator variables , indicating the carriage and carriages Whether the request between passes through the carriage and carriages The connected link interval, if passing through ,otherwise ,Right now: (1); Assume the total link bandwidth is , define the compartment To the target The request response time cost is: (2); in, The response time cost of requesting video content from an external base station for the car; Indicates time period Interior All users in the video content The number of requests; Indicated in Time slot car All users on the video content Whether the request is made by compartment Response, if by carriage Response, then ,otherwise, ; Indicates the carriage and carriages Whether the request between passes through the carriage and goals The link interval connected when it is a carriage; Therefore, during the period Satisfaction Carriage The request response time cost for all users is: (3).

[0040] Step S300 defines the resource replacement time cost according to the difference in cached video content in adjacent time periods; specifically includes: S310. Define binary indicator variables ,express Video content of the period Is it cached in the carriage? On, if The video content of the time period is cached in the carriage On, then ,otherwise, ; S320, in each period At the beginning, if the video content to be cached in the current period is the same as that in the previous period If the cached video content is different, each carriage cache server obtains the video content required for the current period from other carriage cache servers or external base stations to replace the previous period. Cached video content, defined in time periods Time Car The resource replacement time cost is: (4).

[0041] Step S400 formalizes the problem of minimizing the time cost of EMU video service, specifically: Let the EMU video service time cost be the weighted sum of the request response time cost and resource replacement time cost of all carriages in all time periods, and formalize the EMU video service time cost minimization problem as follows: (5); Constraints: (6); (7); (8); (9); (10); in, and is a weight factor used to adjust the weight between request response time cost and resource replacement time cost; Indicates the carriage The upper limit of the cache capacity of the cache server; Indicated in Time slot car All users on the video content Whether the request is made by compartment Response, if the car responds, then ,otherwise, ; Formula (6) ensures that each request is responded to by only one car cache server or external base station. Formula (7) indicates that the cache upper limit of each car cache server should not exceed its own maximum cache capacity limit. Formula (8) ensures that a user request can only be responded to by the car server or external base station that has cached the requested video content. Formulas (9) and (10) limit the indication quantity to a binary indication quantity.

[0042] Step S500 calls a multi-carriage collaborative caching algorithm based on multi-agent reinforcement learning to determine the video resource caching strategy for each time period, such as Figure 4 As shown, including: S510, each carriage is regarded as an intelligent agent. For convenience, the symbol Indicates the carriage Corresponding intelligent agent; using neural network to fit the intelligent agent The action is recorded as the training action network ,in For intelligent agents The training action network parameters of the corresponding target action network are recorded as ; For each agent , set up two training and evaluation networks, respectively denoted as and ,in For intelligent agents The first training evaluation network parameters, the corresponding target network is ,and For intelligent agents The second training evaluation network parameters of the corresponding target network are ; S520, define each intelligent agent exist The status of the time period, including The resource cache status of the carriage itself during the time period, User requests, carriage attributes and The location attributes of the time period; The carriage attribute set is , the location attribute set is , then the agent exist The state of a time period is represented by ,in Representing an Agent The properties of the car you are in. Represents the agent Current location attributes, , indicating the carriage The cache server is The cache status of the time period, Indicates the carriage User in Requests for video content during a time period; S530, according to the obtained intelligent agent The state of the time period, training action network , output each video content The cache probability; since the cached video content cannot exceed the carriage The maximum capacity is 30, by The cache probabilities of the videos are sorted from large to small, and the top 30 video contents are selected as the final cache action; Define the action of the agent as , where if the agent exist Time period cache content ,but ,otherwise, ;and ; S540, each agent Execute an action , the status is Enter the next state ; The environment gives rewards , defined as Time-segment Agent The negative value of the weighted sum of the request response time cost and the resource replacement time cost is expressed as: (11); S550, place all agents in The data of the time period is removed by the time attribute. Stored in the sample pool in the form of five tuples ,in, , represents the global state composed of the local observation states of all agents; , represents the global action composed of the local actions of all agents; , indicating the global state of the next period; , represents the reward set; is a binary indicator variable. If the EMU arrives at the terminal station, ,otherwise, ; S560, if the number of samples reaches the minimum batch processing threshold , train the neural network and update the target network parameters.

[0043] The specific steps for training a neural network are: S561, first calculate the agent The temporal difference error of the two evaluation networks is: (12); (13); in, is the discount factor, For intelligent agents Environmental rewards; Represents the relative Action for the next period; choose Huber error as the final error ,in is the evaluation network number, expressed as: (14); S562, every time Random samples are batch processed and the training and evaluation network parameters are updated by gradient descent, which can be expressed as: (15); in, is the step length; S563: For updating the strategy, gradient ascent is used. To prevent incorrect action evaluation from causing the strategy to be updated in an incorrect direction, the training action network and all target networks are updated with a delay, that is, each update Train the evaluation network again, and then update the training action network and all target networks, expressed as: (16); in, is the step length; For intelligent agents The local observation state.

[0044] S564, using a soft update method to update the parameters of the target network, expressed as: (17); (18); in, , which is used to control the update stride; soft update slowly follows the parameter changes of the training network by gradually adjusting the parameters of the target network.

[0045] Embodiment 2: This embodiment provides a non-transitory computer-readable storage medium, on which computer instructions are stored, and the computer instructions enable the computer to execute a method for collaborative caching of EMU videos based on multi-agent reinforcement learning, the method comprising the following steps: S100. Construct a multi-carriage collaborative caching video service system for EMUs; S200. Define the request response time cost according to the user request cache hit situation; S300 defines the resource replacement time cost according to the difference in cached video content in adjacent time periods; S400. Formalize the time cost minimization problem of EMU video service; S500. Call a multi-carriage collaborative caching algorithm based on multi-agent reinforcement learning to determine the video content caching strategy for each time period.

[0046] Embodiment 3: This embodiment provides an electronic device, which may include: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus. The processor may call the logic instructions in the memory to execute a method for collaborative caching of EMU videos based on multi-agent reinforcement learning, and the method includes the following steps: S100. Construct a multi-carriage collaborative caching video service system for EMUs; S200. Define the request response time cost according to the user request cache hit situation; S300 defines the resource replacement time cost according to the difference in cached video content in adjacent time periods; S400. Formalize the time cost minimization problem of EMU video service; S500. Call a multi-carriage collaborative caching algorithm based on multi-agent reinforcement learning to determine the video content caching strategy for each time period.

[0047] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0048] Embodiment 4: This embodiment provides a computer program product, which includes a computer program. The computer program can be stored in a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute a method for collaborative caching of EMU videos based on multi-agent reinforcement learning. The method includes the following steps: S100. Construct a multi-carriage collaborative caching video service system for EMUs; S200. Define the request response time cost according to the user request cache hit situation; S300 defines the resource replacement time cost according to the difference in cached video content in adjacent time periods; S400. Formalize the time cost minimization problem of EMU video service; S500. Call a multi-carriage collaborative caching algorithm based on multi-agent reinforcement learning to determine the video content caching strategy for each time period.

[0049] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0050] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0051] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for collaborative caching of EMU videos based on multi-agent reinforcement learning, characterized in that: The following steps are involved: S100. Constructing a multi-carriage collaborative caching video service system for EMUs, including defining a set of EMU carriages and a set of video content, and dividing the EMU running time into time periods according to the travel time between stations; S200. Define the request response time cost according to the user request cache hit, including the response within the car, the response between cars and the external base station response; S300 defines the resource replacement time cost according to the difference in cached video content in adjacent time periods; S400. Formalize the time cost minimization problem of EMU video service; S500. Calling a multi-carriage collaborative caching algorithm based on multi-agent reinforcement learning to determine the video content caching strategy for each period, wherein each car is regarded as an agent, a neural network is used to fit the action of the agent, and two evaluation networks are set for collaborative optimization; The data of all agents in a period are stored in the sample pool in the form of five-tuples, including the global state, global action, global state of the next period, reward set and end mark; if the number of samples reaches the minimum batch processing threshold, the neural network is trained and the target network parameters are updated.

2. The EMU video collaborative caching method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The step S100 of constructing a multi-carriage collaborative caching video service system for a train set includes: S110, define the EMU carriage set and video content set, and number the carriages from the front to the rear ,set up Represents a collection of EMU carriages, Represents a collection of video content; S120, divide the EMU running time according to stations, and each period Represents the travel time between the current station and the next station; S130, constructing EMU video service includes: S131. Set up a carriage cache server as a reverse proxy in each carriage, and define the upper limit of the cache capacity of each carriage cache server as , Each carriage cache server is connected to the external base station through the roof antenna of the carriage where it is located. To communicate; users connect to the train LAN through the wireless access points set up in each carriage; the train sets up a central server to query the deployment of the train video content; S132, when the carriage When a user in the train accesses the network and requests video content, the user requests a cache hit, and the user is sent to the user from the train, other trains or an external base station; S133, Order period Interior All users in the video content The number of requests is , define a binary indicator variable , indicating that Time slot car All users on the video content Whether the request is made by the target Response, if by the target Response, then ,otherwise, ,in , Cache server or external base station for EMU carriages.

3. The EMU video collaborative caching method based on multi-agent reinforcement learning according to claim 2 is characterized in that: Step S132: When a user in the train accesses the network and requests video content, the user requests a cache hit, and the user's requested video content is sent to the user from the train, other trains or external base stations; the specific method is: S132-a, determining whether the video content requested by the user is cached in the car where the user is located, if so, executing step S132-e, otherwise executing step S132-b; S132-b, obtaining the deployment status of the video content requested by the user in the EMU from the central server, if the EMU has a cache of the video content requested by the user, executing step S132-c, otherwise executing step S132-d; S132-c, select the car that is closest to the car that issued the request and has cached the video content requested by the user to add to the candidate response car set, check whether there are two optional candidate cars with the same distance, if not, directly select the car that is closest to the car that issued the request and has cached the video content requested by the user as the response car, if so, select the car with the smaller load among the two candidate cars as the response car, and the car where the user is located acts as a proxy to request the video content from the response car, and execute step S132-e after receiving the response; S132-d, the carriage where the user is located acts as an agent to request the video content required by the user from the external base station, and after receiving the response, executes step S132-e; S132-e. The carriage where the user is located sends the video content requested by the user to the users in the same carriage.

4. The EMU video collaborative caching method based on multi-agent reinforcement learning according to claim 3 is characterized in that: Step S200 defines the request response time cost according to the cache hit situation of the user request; specifically includes: According to the cache hit situation of the user request, there are three different responses, namely, in-car response, inter-car response and external base station response, corresponding to three different request response time costs; Define binary indicator variables , indicating the carriage and carriages Whether the request between passes through the carriage and carriages If the connected link interval passes through ,otherwise ,Right now: （1）； Assume the total link bandwidth is , define the compartment To the target The request response time cost is: （2）； in, The response time cost of the car requesting video content from the external base station is set to a fixed value; Indicates time period Interior All users in the video content Number of requests; Indicated in Time slot car All users on the video content Whether the request is made by compartment Response, if by carriage Response, then ,otherwise, ; Indicates the carriage and carriages Whether the request between passes through the carriage and goals The link interval connected when it is a carriage; In the period Satisfaction Carriage The request response time cost for all users is: （3）。 5. The method for collaborative caching of train group videos based on multi-agent reinforcement learning according to claim 4 is characterized in that: Step S300 defines the resource replacement time cost according to the difference in cached video content in adjacent time periods; specifically includes: S310. Define binary indicator variables ,express Video content of the period Is it cached in the carriage? On, if The video content of the time period is cached in the carriage On, then ,otherwise, ; S320, in each period At the beginning, if the video content to be cached in the current period is the same as that in the previous period If the cached video content is different, each carriage cache server obtains the video content required for the current period from other carriage cache servers or external base stations to replace the previous period. Cached video content, defined in time periods Time Car The resource replacement time cost is: （4）。 6. The EMU video collaborative caching method based on multi-agent reinforcement learning according to claim 5 is characterized in that: Step S400 formalizes the problem of minimizing the time cost of EMU video service, specifically: Let the EMU video service time cost be the weighted sum of the request response time cost and resource replacement time cost of all carriages in all time periods, and formalize the EMU video service time cost minimization problem as follows: （5）； Constraints: （6）； （7）； （8）； （9）； （10）； in, and is a weight factor used to adjust the weight between request response time cost and resource replacement time cost; Indicates the carriage The upper limit of the cache capacity of the cache server; Indicated in Time slot car All users on the video content Whether the request is made by compartment Response, if by carriage Response, then ,otherwise, .

7. The EMU video collaborative caching method based on multi-agent reinforcement learning according to claim 6 is characterized in that: Step S500 includes: S510, each carriage is regarded as an intelligent agent, and the carriage number is Represents the intelligent agent corresponding to the carriage; uses a neural network to fit the intelligent agent The action is recorded as the training action network ,in For intelligent agents The training action network parameters of the corresponding target action network are recorded as ; For each agent , set up two training and evaluation networks, respectively denoted as and ,in For intelligent agents The first training evaluation network parameters, the corresponding target network is ,and For intelligent agents The second training evaluation network parameters of the corresponding target network are ; S520, define each intelligent agent In the period status, including The resource cache status of the carriage itself during the time period, User requests, carriage attributes and The location attributes of the time period; The carriage attribute set is , the location attribute set is , then the agent exist The state of a time period is represented by ,in Representing an Agent The properties of the car you are in. Represents the agent Current location attributes, , indicating the carriage The cache server is The cache status of the time period, Indicates the carriage User in Requests for video content during a time period; S530, according to the obtained intelligent agent The state of the time period, training action network , output each video content The cache probability of each video content Sort the cache probabilities from large to small and select the first The video content is used as the final cache action; Define the action of the agent as , where if the agent exist Time period cache content ,but ,otherwise, ;and ; S540, each agent Execute an action , the status is Enter the next state ; The environment gives rewards , defined as Time-segment Agent The negative value of the weighted sum of the request response time cost and the resource replacement time cost is expressed as: （11）； S550, place all agents in The data of the time period is removed by the time attribute. Stored in the sample pool in the form of five tuples ,in, , represents the global state composed of the local observation states of all agents; , represents the global action composed of the local actions of all agents; , indicating the global state of the next period; , represents the reward set; is a binary indicator variable. If the EMU arrives at the terminal station, ,otherwise, ; S560, if the number of samples reaches the minimum batch processing threshold , train the neural network and update the target network parameters.

8. The method for collaborative caching of train group videos based on multi-agent reinforcement learning according to claim 7 is characterized in that: Step S560 is to train the neural network, and the specific steps are as follows: S561, first calculate the agent The temporal difference error of the two evaluation networks is: （12）； （13）； in, is the discount factor; For intelligent agents Rewards; Represents the relative Action for the next period; Huber error is selected as the final error ,in is the evaluation network number, expressed as: （14）； S562, every time Random samples are batch processed and the training and evaluation network parameters are updated by gradient descent, which can be expressed as: （15）； in, is the step length; S563, for the update of the strategy, gradient ascent is used; for the training action network and all target networks, the update is delayed, that is, each update Train the evaluation network again, and then update the training action network and all target networks, expressed as: （16）； in, is the step length; For intelligent agents The local observation state of S564, using a soft update method to update the parameters of the target network, expressed as: （17）； （18）； in, is a coefficient between 0 and 1 that controls the step size of the update; soft updates slowly follow the parameter changes of the training network by gradually adjusting the parameters of the target network.

Images