Dynamic multimedia data deployment method based on reinforcement learning

Abstract

The application discloses a dynamic multimedia data deployment method based on reinforcement learning, comprising the following steps: establishing a meta-learning model and acquiring dynamic multimedia data; a meta-pretraining process: selecting days N to be meta-pretrained, days i for online meta-adaptation and a task environment sequence to be dynamically adapted from the dynamic multimedia data; selecting initial model parameters for the task environment in different sequences; inputting state variables, training according to a loss function of the meta-learning model after generating actions and rewards, and obtaining dynamic meta-knowledge; an online meta-adaptation process: inputting a new dynamic task environment, initializing a model by using the dynamic meta-knowledge; updating the meta-learning model on the new task environment, performing generalization training of the dynamic meta-knowledge, and obtaining a new model and meta-knowledge. The application can avoid the performance decline of edge content cache hit rate caused by a dynamic request mode under the constantly changing video popularity.

Description

Technical Field

[0001] This invention relates to the field of multimedia data deployment technology, and in particular to a dynamic multimedia data deployment method based on reinforcement learning. Background Technology

[0002] In recent years, we have witnessed the rapid development and construction of edge-assisted content delivery infrastructure. Video streaming has experienced explosive growth in recent years. Moving content replication and distribution modules to the edge of the internet (or network) has the potential to reduce the workload of the backbone network and improve the quality of experience (QoE) for video streaming users. This type of video content service is often referred to as edge content delivery. Although the exact name is still debated in the industry, edge content delivery has undergone a fundamental change.

[0003] With the rapid evolution of video platforms, request patterns in edge servers are more dynamic and distributed compared to traditional methods. For example, a measurement study of Kuaishou, one of the most well-known short video sharing platforms, shows that the difference in content request patterns across days measured by Kullback-Leibler (KL) divergence increased by 35.1% compared to traditional video, from 23.30 to 31.47 based on the sharing platform. This indicates that content popularity varies significantly over time, especially for short video platforms. This is primarily because edge caching is typically dedicated to only a small subset of users.

[0004] This dynamic request pattern poses challenges to existing content caching methods in edge content distribution: 1) Traditional schemes with prior assumptions, including Least Recently Used (LRU), Least Frequently Used (LFU), and their variants, struggle to adapt to dynamic environments in a timely manner because they are based on simple rule-based policies and some manually set parameter characteristics. 2) Reinforcement learning (RL)-based caching strategies have been proven to achieve higher hit rates than traditional rule-based strategies. However, their design assumptions are based on static and stable environments, which contradicts the dynamic nature of current content patterns, as illustrated by relevant measurements. When applied to dynamic edge content distribution, this assumption cannot be guaranteed, often leading to performance degradation. One possible reason is that dynamic request patterns gradually weaken static assumptions, causing the gap between the assumptions and real-world request patterns to widen. Therefore, outdated historical data continuously degrades the real-time performance of reinforcement learning models. 3) Improvements to reinforcement learning-based methods, including those using manually selected features and others using recursive network architectures to extract dynamic features, are still affected by rapidly changing environments. Summary of the Invention

[0005] The purpose of this invention is to solve the problem of enabling edge content distribution to adapt to dynamic edge environments more quickly while ensuring performance, and to provide a dynamic multimedia data deployment method based on reinforcement learning.

[0006] The technical problem of this invention is solved by the following technical solution:

[0007] A reinforcement learning-based method for dynamic multimedia data deployment includes the following steps:

[0008] S0. Establish a meta-learning model and acquire dynamic multimedia data;

[0009] S1, Meta-pre-training process:

[0010] S11. Select from the dynamic multimedia data the number of days N for meta-pre-training, the number of days i for online meta-adaptation, and the task environment sequence T1 to T2 for dynamic adaptation. N+i ;

[0011] S12. Select initialization model parameters for the task environment for different sequences;

[0012] S13. Input state variables, generate actions and rewards, and then train the meta-learning model according to its loss function to obtain dynamic meta-knowledge;

[0013] S2, Online Meta-Adaptive Process:

[0014] S21. Input the new dynamic task environment T N+i+1 The model is initialized using the dynamic meta-knowledge.

[0015] S22. Update the meta-learning model in the new task environment, and at the same time perform generalization training of dynamic meta-knowledge to obtain a new model and meta-knowledge.

[0016] In some embodiments, step S1 includes: jointly updating the model meta-knowledge parameters θ and the learning rate α using the historical request sequence to obtain initial meta-knowledge.

[0017] In some embodiments, step S12 includes: using task T i The model parameters θ are adapted to obtain the model update parameters φ to collect data for task T. i+1 The trajectory.

[0018] In some embodiments, step S13 includes: using content request data from adjacent time series to perform meta-learning-based two-layer optimization training to obtain transferable information in adjacent tasks.

[0019] In some embodiments, the loss function of the meta-learning model described in step S13 is expressed by the following formula:

[0020]

[0021] Where τ refers to the state state t action t Reward R t state (t+1) The trajectory is composed of the model parameter φ, the meta-knowledge parameter θ, the learning rate α, and the task environment T. i 1:K indicates that there are K trajectories, M indicates the number of update steps, and P() indicates that the trajectories follow a data distribution with probability P.

[0022] In some embodiments, step S21 includes: after the initial meta-knowledge of the N-day task is provided in the meta-pre-training process, new meta-knowledge θ of N+i days is searched so as to achieve a better adaptation strategy within only a few samples after encountering a new dynamic task.

[0023] In some embodiments, step S22 includes: finding some general features as meta-knowledge and obtaining transferable meta-knowledge θ to help the model obtain better adaptation rules; and achieving rapid adjustment through optimization based on re-initialized proximal policies, which is controlled by the online acquired meta-knowledge.

[0024] In some embodiments, in step S22, the Regret function of the meta-learning model is used for generalization training of dynamic meta-knowledge.

[0025] In some embodiments, step S2 uses an online approach to update meta-knowledge and maintain its long-term impact for online meta-adaptation. The meta-knowledge is refreshed and updated through multi-step online gradient descent to find some general features and obtain transferable meta-knowledge to help the agent obtain better adaptation rules.

[0026] The present invention also provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the above-described reinforcement learning-based dynamic multimedia data deployment method.

[0027] The present invention has the following beneficial effects:

[0028] This invention captures dynamically changing sequence pair meta-knowledge through a meta-pre-training process and an online meta-adaptation process, and keeps the meta-knowledge updated online, enabling the model to adapt to edge dynamic environments more quickly. At the same time, by avoiding dynamic request patterns, this invention can avoid the performance degradation of edge content cache hit rate caused by dynamic request patterns under constantly changing video popularity, thus enabling the model to adapt to edge dynamic environments more quickly. Attached Figure Description

[0029] Figure 1This is a schematic diagram of a caching strategy based on reinforcement learning (RL) in the existing technology;

[0030] Figure 2 This is a flowchart of a dynamic multimedia data deployment method based on online meta-reinforcement learning in an embodiment of the present invention;

[0031] Figure 3 This is a schematic diagram of the meta-pre-training process and the online meta-adaptive process in an embodiment of the present invention;

[0032] Figure 4 This is a schematic diagram of a dynamic multimedia data deployment method based on online meta-reinforcement learning in an embodiment of the present invention;

[0033] Figure 5 This is a schematic diagram illustrating the phenomenon of hit rate changes of existing technologies in a dynamic environment on a long video platform in the experimental example;

[0034] Figure 6 This is a schematic diagram illustrating the phenomenon of hit rate changes of existing technologies in the dynamic environment of short video platforms in the experimental example;

[0035] Figure 7 This is a schematic diagram illustrating the phenomenon of hit rate changes in a dynamic environment on a long video platform in an experimental example of the present invention.

[0036] Figure 8 This is a schematic diagram illustrating the phenomenon of hit rate changes in the dynamic environment of a short video platform in an experimental example of the present invention. Detailed Implementation

[0037] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and not intended to limit the scope and application of the present invention.

[0038] Example

[0039] To address the challenge of enabling edge content distribution to adapt more quickly to dynamic edge environments while maintaining performance, a direct solution is to restart the reinforcement learning model to overcome the long-term effects of outdated data. However, reinforcement learning models require a long adaptation time to achieve good performance. To achieve faster adaptation, it's worth considering providing the reinforcement learning model with some transferable knowledge when handling new dynamic tasks. Meta-reinforcement learning shows promise as a solution to these challenges, as it has been shown to achieve significant adaptation to new environments in previous work. However, it has been found that in this case, simply using meta-reinforcement learning can even outperform ordinary reinforcement learning methods. This is because ordinary meta-reinforcement learning ignores dynamic features in the time domain. Meta-reinforcement learning tasks are designed for multi-task adaptation and lack time-series design, while caching problems are clearly time-series ordered, making them unsuitable for edge caching problems.

[0040] To enable meta-reinforcement learning to adapt to such dynamic environments, this embodiment proposes a dynamic multimedia data deployment strategy based on online meta-reinforcement learning, with two key design goals: 1) meta-learning should capture time-transferable knowledge to adapt to dynamic environments; 2) meta-knowledge should be continuously updated to maintain timeliness. To achieve 1), meta-reinforcement learning is reformulated by segmenting dynamic episodes into a series of short intervals, and then dynamically changing knowledge is acquired from sequence pairs. For 2), an online meta-learning paradigm based on online gradient descent (OGD) is proposed to maintain the freshness of meta-reinforcement learning knowledge.

[0041] The model portion of the embodiments of the present invention for achieving the above objectives is described below:

[0042] Online Sequence Pair Meta-Reinforcement Learning (OMAC) consists of two main parts: (1) meta-reinforcement learning in the form of sequence pairs and (2) online updates.

[0043] Part One:

[0044] Meta-reinforcement learning in sequence pair form can be decomposed into two sub-concepts: (1.1) reinforcement learning and (1.2) meta-learning in sequence pair form.

[0045] refer to Figure 1 The empirical loss function of the existing technology model is l T The optimization objective of reinforcement learning is given by the following formula (φ), where the parameter φ is the parameter of the reinforcement learning model:

[0046] (This is based on reinforcement learning, which involves taking the expectation of the model parameters in an environment T.) Figure 1 φ i Ti is a specific one of φ, and T is a specific one of T. Model φ i In the Ti environment assessment, the empirical loss function is l Ti (φ i In the cache, the model, or agent, optimizes possible decisions based on past experience and gained rewards. Specifically, after observing the state (state variable), an agent takes an action in the current task environment T and receives a corresponding reward R, where R is a common discounted cumulative reward. Where W is a time window related to the sequence length of the LSTM; r tThe values ​​are related to R, where R is the gain the model obtains in the Markov process over a certain period, and rt is the gain obtained at one of the time steps. γ is a constant less than 1, which decreases continuously with the power of time; in this embodiment, it is taken as 0.99. This forms a standard Markov process: each state transition depends only on the previous n states. This process is called an n-order model, where n is the number of states affecting the transition. The simplest Markov process is a first-order process, where each state transition depends only on its previous state, also known as the Markov property. In the edge caching problem, this state is the cached content from the previous time step combined with the current request status. t Then it is triggered by an action. t The effect produces the next state. (t+1) .

[0047] Then, meta-learning is a two-layer optimization form of reinforcement learning (two-layer optimization means that, based on this, in the same environment T, the model parameters adapted over a period of time are added again, and the expectation is taken again; note that the environment T for the two expectations is the same, that is, E in the above formula needs to be performed again). This embodiment also makes certain improvements to this two-layer optimization form. The two layers are divided into an inner loop and an outer loop. The inner loop parameter φ runs in task T. i+1 Above, let the outer loop parameter θ run in task T. i The above (φ is obtained by updating θ through gradient descent). This allows this embodiment to measure the adaptation effect of meta-knowledge θ in such a form, thereby improving the model performance. The optimization objective of meta-learning is as follows:

[0048]

[0049] The optimization objective, i.e., the loss function, for meta-learning in the form of sequence pairs is given by the following formula:

[0050]

[0051] Where τ refers to the state state t action t Reward R t state (t+1) The trajectory is composed of the model parameter φ, the meta-knowledge parameter θ, the learning rate α, and the task environment T. i1:K indicates that there are K trajectories, M indicates the number of update steps, ~P() indicates that the trajectory follows a data distribution with probability P, N refers to the number of days extracted from all the days of dynamic multimedia data for meta-pre-training, this part of the data only participates in pre-training and not in online detection; i is the total number of days used for online meta-adaptation.

[0052] The above improvements take into account the dynamic nature of edge environment data distribution. Not only do the model parameters update and change over time, but the environment T also changes continuously over time. The model not only needs to learn its own changes, but also its ability to adapt to new environments.

[0053] Part Two:

[0054] Considering that meta-knowledge is not updated over time, this embodiment implements online updates for meta-learning. This update is necessary because, due to the requirements of real-world scenarios, meta-knowledge θ cannot remain static; otherwise, the model's performance would deteriorate over time because the data distribution at that point in time differs from the data distribution when the meta-knowledge was obtained. Therefore, the meta-knowledge θ is updated daily using the following formula, i.e., the Regret function of the meta-learning model performs dynamic meta-knowledge generalization training:

[0055]

[0056] In some embodiments, other functions can also be used for generalization training of dynamic meta-knowledge.

[0057] Based on the above online sequence pair meta-framework, this embodiment proposes a practical method for dynamic multimedia data deployment using online meta-reinforcement learning, such as... Figure 2 , Figure 3 As shown, this method consists of two phases: meta-learning and online meta-adaptation, including the following steps:

[0058] S0. Establish a meta-learning model and acquire dynamic multimedia data;

[0059] S1, Meta-pre-training process:

[0060] S11. Select the number of days N for meta-pre-training, the number of days i for online meta-adaptation, and the task environment sequence T1 to T2 for dynamic adaptation from the dynamic multimedia data. N+iN refers to the number of days extracted from all days of dynamic multimedia data for meta-pre-training. This part of the data only participates in pre-training and not in online detection. i is the total number of days used for online meta-adaptation. This part of the data starts updating the model and detecting the final model hit-rate metric as soon as the first request arrives.

[0061] S12. Select initialization model parameters for the task environment for different sequences;

[0062] S13. Input state variables, generate actions and rewards, and then train the meta-learning model according to its loss function to obtain dynamic meta-knowledge;

[0063] S2, Online Meta-Adaptive Process:

[0064] S21. Input the new dynamic task environment T N+i+1 The model is initialized using the dynamic meta-knowledge.

[0065] S22. Use gradient descent to update the meta-learning model in the new task environment, and use the Regret function of the meta-learning model to perform dynamic meta-knowledge generalization training to obtain a new model and meta-knowledge.

[0066] In this embodiment, the meta-learning model employs a common reinforcement learning actor-critic network, and the preprocessed features utilize an LSTM deep learning model. Initial meta-knowledge refers to the ungeneralized features obtained during the meta-pre-training process, while dynamic meta-knowledge refers to the generalized features based on sequence pairs obtained during the meta-pre-training and online meta-adaptation processes. In this embodiment, the selection criterion for N is time segments with semantic meaning that do not repeat real-world scenarios. For example, for a model based on weekdays, N is 5 instead of 10 or 15; for a model based on natural weeks, N is 7 instead of 14. After determining N, as many i as possible are collected.

[0067] For details, please refer to Figure 4 This embodiment proposes a practical method for deploying dynamic multimedia data using online meta-reinforcement learning, including:

[0068] 1. Meta-pre-training process (acquiring meta-knowledge through sequence pairing tasks): First, based on the content request data of the original system, a meta-pre-training process is performed using a sequence pairing meta-loss function designed for dynamic caching task meta-knowledge (acquiring meta-knowledge through sequence pairing tasks). A small number of historical request sequences are used to jointly update and obtain the initial meta-knowledge θ. i In the meta-pre-training part, a small number of historical request sequences are used to jointly update the model's meta-knowledge parameters θ and learning rate α to obtain the initial meta-knowledge θ for the subsequent i specific task environments. i .

[0069] To obtain dynamic meta-knowledge with generalization performance again, θ i In the task environment T i Continuing with dynamic feature learning, using data from task T... i The model parameters θ are adapted to obtain the model update parameters φ to collect data for task T. i+1 The trajectory. That is, meta-knowledge θ. i Current environment T i By using a step size 'a' for learning, a more adaptive model 'φ' was obtained. i This adaptive model can be applied to environment T. i+1 An evaluation is conducted on T during this process. i The trajectory may contain some suitable for T i+1 The transferable information is because the sequential pairing task depends on each adjacent task.

[0070] In this embodiment, the sequence pair design provides for dynamic changes in initial meta-knowledge; online updates help maintain the effectiveness of meta-knowledge, enabling this embodiment to handle dynamic caching.

[0071] 2. Define the state space, action space, reward function, etc., in the definition of the edge cache content distribution problem. Use content request data from adjacent time series to perform two-layer optimization training based on meta-learning, through A... GAE (Generalized Advantage Estimation) Obtains transferable information in adjacent tasks.

[0072] This embodiment of the system assumes that each cache is at most S in size, and caches data F consisting of C equally sized content files. At each time step, there is a request for f belonging to F, denoted as x^t. The cache size on each node is denoted as c. The model input state is State. t Including current content requests and caching status ( After inputting the state, the model gives an action in [1, S+1] according to its policy.

[0073] If the action output is S+1, the proxy will not perform content replacement. Otherwise, when the action output is i, the cached content at position i will be replaced. The newly arrived content ID also applies to the adaptation process, as it only involves the cache and the state of the incoming request, not a specific ID. After this operation, the proxy receives a reward for whether the cached content was hit, and the state changes to State. (t+1) .

[0074] 3. Online Meta-Adaptive Process (Update Meta-Knowledge):

[0075] The method uses online meta-knowledge updates and maintains its long-term impact for online meta-adaptation (refreshing meta-knowledge). The goal of updating meta-knowledge through multi-step online gradient descent (OGD) is to find some general features and obtain transferable meta-knowledge to help the agent obtain better adaptation rules.

[0076] Intuitively, after the meta-pre-training process provides initial meta-knowledge for N days of tasks, the framework's online meta-adaptation part searches for new meta-knowledge θ for N+i days to achieve a better adaptation strategy within just a few samples when encountering new dynamic tasks. This OGD-like iteration guarantees sublinear Regret, thus ensuring that the online meta-knowledge is close to optimal.

[0077] The goal of the meta-learning phase is to find some general features as meta-knowledge and obtain transferable meta-knowledge θ to help the model acquire better adaptation rules. This strategy is rapidly tuned through optimization based on re-initialized proximal policies (using AGAE as the loss function), which is controlled by online-acquired meta-knowledge.

[0078] This embodiment discovers the dynamic characteristics of edge content popularity, proving that it leads to a performance degradation of ordinary reinforcement learning. It inspires the use of meta-learning methods, proposing that ordinary meta-learning cannot improve the performance of reinforcement learning-based algorithms. A new framework, called the Online Sequence Pair Meta-Learning (OMAC) framework, is developed to handle dynamic caching: 1) the sequence pair design provides initial meta-knowledge that changes dynamically; 2) online updates help maintain the validity of meta-knowledge. Based on this framework, a dynamic multimedia data deployment method based on online meta-reinforcement learning is proposed. This method enables deep learning models to generalize more quickly and adapt to edge data distribution in more dynamic and non-stationary data distribution environments at the edge, allowing decision models to provide timely and effective services. Furthermore, in content distribution networks, based on the popularity characteristics of content in different regions, the content that users are interested in is estimated, and the content is pre-distributed to servers that can provide a better experience, reducing user waiting latency.

[0079] The advantage of this embodiment is that:

[0080] 1. Accurately model the dynamic request distribution on edge content caching as a Markov decision process to improve the generalization ability of model-based methods.

[0081] 2. By making full use of current information through gradient estimation that relies on prior historical knowledge, edge caching model training has better convergence performance.

[0082] 3. Avoid using heuristics designed by hand. The meta-knowledge obtained is applicable to the dynamic situation of edge caching environment, with better generalization performance and content hit rate results.

[0083] Experimental Example

[0084] This experimental example uses real data records (anonymized user information) from iQiyi and Kuaishou for comprehensive testing. The results show that the proposed method consistently produces better optimization results than the baseline. Specifically, it achieves best performance in all 56 randomly selected edge regions, and achieves an average hit rate improvement of 16.3% and 37.4% in iQiyi and Kuaishou tracking, respectively. To evaluate robustness, this experimental example is evaluated for different cache sizes and edge service regions. The results show that the proposed method maintains best performance in every case. Ablation studies also show that both the sequence pair design step and the online update step contribute to improved performance.

[0085] The specific process is as follows:

[0086] 1. Data Description:

[0087] Data was collected from content request records across iQiyi and Kuaishou platforms over 13 days, with timestamps and locations extracted for each request. The iQiyi data records contained 53,954,230 requests for 417,077 video content items, while the Kuaishou data records contained 52,852,160 requests for 1,746,227 video content items. Since tracking analysis showed different request patterns between regions, this experiment randomly selected 56 representative edge regions (2.05km * 2.31km) as the represented edge regions to verify the robustness of the method. Furthermore, the video content distribution range of the edge caching devices was distributed sequentially from 0.59km^2 to 303.07km^2 to evaluate the impact of service range. In the experiment, it was assumed that the content delivery network edge cache located at the center of the edge region served the requests for that region.

[0088] 2. Experimental Procedure: First, the required number of days N for the meta-pre-training process was selected, along with a fixed cache size and a fixed service area. Then, a small number of historical request sequences were used to jointly update the model's meta-knowledge parameters θ and learning rate α to obtain initial meta-knowledge. From task T... i The model parameters θ are adapted to obtain the model update parameters φ to collect data for task T. i+1 The trajectory of T. i The trajectory may contain some suitable for T i+1 The transferable information is because the sequential pairing task depends on each adjacent task.

[0089] Then, based on the initial meta-knowledge, the subsequent online meta-adaptation process is carried out. The online meta-adaptation part will search for new meta-knowledge θ for N+i days so as to achieve a better adaptation strategy within only a few samples after encountering new dynamic tasks.

[0090] Then, the experiment was repeated by changing hyperparameters such as cache size and service area to prove the robustness of the results.

[0091] 3. Specific parameters:

[0092] The default cache capacity for each edge is set to 0.02% of the total number of requested video content. This experimental example uses TensorFlow to implement an online reinforcement meta-learning model framework, which runs on a server with dual RTX 2070 GPUs and 6GB of memory. The learning rates for meta-learning and meta-adaptation are set to 1e-3 and 2e-4, respectively. The default reward discount factor for reinforcement learning is set to 0.99. The cached states over the past 7 time steps are considered as the number of states retained by the LSTM. Furthermore, the default number of days for meta-pre-training is set to 5, MAML is executed 3 times per adaptation round, and meta-knowledge is updated daily.

[0093] 4. Experimental Results:

[0094] Figure 5 , Figure 6 The performance spikes observed represent the characteristics of cached data distribution learned by the model in a specific environment. When the environment is fixed, the more features the model learns over time, the higher the hit rate. However, because the edge environment is dynamic, changes occur after a period of time, causing the data distribution to shift from the previously learned features. Due to the non-independent and identically distributed data, the model's performance deteriorates, failing to respond well to current content requests, resulting in a sharp drop in caching effectiveness. (Reference) Figure 7 , Figure 8 For experiments involving dynamic adaptation, the model in this embodiment of the invention can achieve a high hit rate in a short time without significant spikes or subsequent performance degradation. Figure 7 , Figure 8The phenomena observed indicate that if the model using the embodiments of the present invention learns more about the adaptive features of environment switching for sequence pairs, it can ensure a high hit rate even when the environment is dynamically changing. This demonstrates that the method of the embodiments of the present invention can continuously adapt to constantly changing environments and maintain good performance. This is because other methods, which rely on assumptions of prior knowledge or stationarity, may perform well initially. However, when the inconsistency between the assumptions and the environment becomes more significant, it leads to a severe performance degradation. The sequence pair meta-learning framework of the embodiments of the present invention can capture meta-knowledge of environmental changes, which helps the re-initialized model adapt quickly. Based on this framework, the meta-knowledge is updated online to maintain its freshness and continuously provides real-time adaptation rules, resulting in the final online sequence pair meta-learning framework; therefore, the method of the embodiments of the present invention is not affected by the performance degradation caused by changes in content popularity. For robustness experiments, the model of the embodiments of the present invention achieves a high hit rate under different edge cache sizes and different service area sizes. In this experimental example, the content cache size is gradually increased from 0.02% to 0.10% of the total number of requested video content, and the service area is gradually increased from 0.59 km² to 303.07 km². The hit rate metrics of the methods in this invention are higher than those of traditional statistical and reinforcement learning methods. This invention achieves state-of-the-art performance on both short video and traditional video sharing platforms, which helps the model perform well on the increasingly dynamic short video platform.

[0095] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0096] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0097] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0098] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0099] The above description provides a further detailed explanation of the present invention in conjunction with specific / preferred embodiments, and it should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various substitutions or modifications can be made to these described embodiments without departing from the concept of the present invention, and all such substitutions or modifications should be considered within the scope of protection of the present invention. In the description of this specification, the reference to terms such as "an embodiment," "some embodiments," "preferred embodiment," "example," "specific example," or "some examples," etc., indicates that the specific features, structures, materials, or characteristics described in connection with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples. Although the embodiments of the present invention and their advantages have been described in detail, it should be understood that various changes, substitutions, and modifications can be made herein without departing from the scope of protection of the patent application.

Claims

1. A dynamic multimedia data deployment method based on reinforcement learning, characterized in that, Includes the following steps: S0. Establish a meta-learning model and acquire dynamic multimedia data; S1, Meta-pre-training process: S11. Select from the dynamic multimedia data the number of days N for meta-pre-training, the number of days i for online meta-adaptation, and the task environment sequence T1 to T2 for dynamic adaptation. N+i ; S12. Select initialization model parameters for the task environment for different sequences; among them, using the sequence from task T i The model parameters θ are adapted to obtain the model update parameters φ to collect data for task T. i+1 The trajectory; S13. Input state variables, generate actions and rewards, and then train the model according to the loss function of the meta-learning model to obtain dynamic meta-knowledge; the loss function of the meta-learning model is expressed by the following formula: Where τ refers to the state state t action t Reward R t state (t+1) The trajectory is composed of the model parameter φ, the meta-knowledge parameter θ, the learning rate α, and the task environment T. i 1:K indicates that there are K trajectories, M indicates the number of update steps, and P() indicates that the trajectories follow a data distribution with probability P; S2, Online Meta-Adaptive Process: S21. Input the new dynamic task environment T N+i+1 The model is initialized using the dynamic meta-knowledge. S22. Update the meta-learning model in the new task environment, and simultaneously perform generalization training of dynamic meta-knowledge to obtain a new model and meta-knowledge; among which, the Regret function of the meta-learning model is used. Perform generalization training using dynamic meta-knowledge; in this process, find some general features as meta-knowledge and obtain transferable meta-knowledge θ to help the model obtain better adaptation rules; and achieve rapid tuning through optimization based on re-initialized proximal policies, which is controlled by online acquired meta-knowledge. Among them, online meta-knowledge updates are used to maintain its long-term impact for online meta-adaptation. The meta-knowledge is refreshed and updated through multi-step online gradient descent to find some general features and obtain transferable meta-knowledge to help the agent obtain better adaptation rules.

2. The dynamic multimedia data deployment method based on reinforcement learning as described in claim 1, characterized in that, Step S1 includes: jointly updating the model meta-knowledge parameters θ and the learning rate α using the historical request sequence to obtain initial meta-knowledge.

3. The dynamic multimedia data deployment method based on reinforcement learning as described in claim 1, characterized in that, Step S13 includes: using content request data from adjacent time series to perform meta-learning-based two-layer optimization training to obtain transferable information in adjacent tasks.

4. The dynamic multimedia data deployment method based on reinforcement learning as described in claim 1, characterized in that, Step S21 includes: after providing initial meta-knowledge for N days of tasks during the meta-pre-training process, searching for new meta-knowledge θ for N+i days to achieve a better adaptation strategy within just a few samples after encountering new dynamic tasks.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the method as described in any one of claims 1 to 4.

Images