Method and device for autonomous updating of CDN multi-source station back-to-source strategy

Abstract

The application relates to the technical field of communication, and provides a CDN multi-source station back-to-source strategy autonomous updating method and device, which comprises the following steps: acquiring a first source station state vector of each source station, a first user state vector of a user application and a node state vector of each CDN edge node in real time, and performing normalization processing; inputting the normalized first source station state vector, the first user state vector and the node state vector into a back-to-source strategy autonomous updating model to obtain a back-to-source autonomous updating action representing the load proportion of each source station output by the back-to-source strategy autonomous updating model; and issuing the back-to-source autonomous updating action to each CDN edge node, wherein the back-to-source autonomous updating action is used for instructing the CDN edge node to perform back-to-source based on the back-to-source autonomous updating action and perform back-to-source according to the updated back-to-source strategy. The application determines the back-to-source strategy based on the real-time states of each source station, the user application and each CDN edge node, so that the CDN multi-source station back-to-source accuracy is high, and the timeliness is stronger.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to a method and apparatus for autonomously updating CDN multi-source station back-to-origin strategy. Background Technology

[0002] Content Delivery Network (CDN) origin pull refers to the process of forwarding user requests to the origin server to retrieve content when CDN edge nodes cannot hit the cache. Currently, CDN multi-origin origin pull strategies mainly use a round-robin approach, supporting multiple origin servers and multiple IPs, allowing for quick switching from one origin server to another. Configuration supports different probe times and probe counts. Probes are performed using methods such as ping or GET, and if the primary origin server encounters problems or is unavailable, the system switches to a backup origin server. The entire process is automated once configured. A 1:1:1 ratio can be configured for origin pull (other ratios can be set, such as 1:2:3, which represents the proportion of resources retrieved by the edge node from multiple origin servers). If the probe server detects a particular origin server and fails three times (the strategy is adjustable), it actively switches to another origin server.

[0003] The aforementioned method of configuring origin strategies based on manual experience is difficult to automatically switch and adjust the allocation ratio in real time according to the situation of each origin station. This results in problems such as low origin accuracy (and the origin station not having sufficient resources) and poor timeliness. Summary of the Invention

[0004] This invention provides a method and apparatus for autonomously updating CDN multi-origin site back-to-origin strategy, which solves the problems of low back-to-origin accuracy and poor timeliness in the prior art when CDN multi-origin site back-to-origin.

[0005] This invention provides a method for autonomously updating CDN multi-origin site back-to-origin strategy, including the following steps.

[0006] The system acquires the first source station state vector, the first user state vector of the user application, and the node state vector of each CDN edge node in real time, and performs normalization processing on the first source station state vector, the first user state vector, and the node state vector respectively.

[0007] The normalized first source station state vector, first user state vector, and node state vector are input into the source return strategy autonomous update model to obtain the source return autonomous update action representing the load ratio of each source station output by the source return strategy autonomous update model.

[0008] The origin self-update action is sent to each CDN edge node. The origin self-update action is used to instruct the CDN edge node to perform origin recursion with the updated origin recursion strategy based on the origin self-update action.

[0009] The source-following strategy autonomous update model is a deep reinforcement learning-based model, trained using (s,a,c,r,s') as training samples and the target Q-value Q(s,a,c) corresponding to (s,a,c,r,s') as the label. Here, s is the set of the first source station state sample vector and the first user state sample vector at the current time step, c is the node state sample vector at the current time step, r is the current reward value of the source-following strategy autonomous update model, s' is the set of the first source station state sample vector and the first user state sample vector at the next time step, and a is the source-following autonomous update action sample corresponding to s at the current time step.

[0010] The CDN multi-origin site back-to-origin strategy self-update method provided by the present invention further includes the following steps.

[0011] After obtaining the second source station state vector of each source station and the second user state vector of the user application, the data is retrieved.

[0012] Calculate the source station state difference between the second source station state vector and the first source station state vector, and calculate the user state difference between the second user state vector and the first user state vector.

[0013] The source station state difference and the user state difference are input into the reward neural network to obtain the latest reward value output by the reward neural network, and the current reward value is updated with the latest reward value.

[0014] The reward neural network is trained using the time series difference between the source station's historical state data and the time series difference between the corresponding user's historical state data as samples, and the reward truth value corresponding to the time series difference between the source station's historical state data as the label.

[0015] According to the present invention, a CDN multi-origin site back-to-origin strategy autonomous update method is provided, wherein the reward neural network includes: an origin site state difference feature extraction module, a user state difference feature extraction module, and an attention layer module, wherein the origin site state difference feature extraction module and the user state difference feature extraction module are both connected to the attention layer module.

[0016] The source station status difference feature extraction module is used to extract the real-time status difference feature vector of each source station based on the source station status difference of each source station.

[0017] The user state difference feature extraction module is used to extract the real-time state difference feature vector of the user based on the user state difference.

[0018] The attention layer module is used to predict and output the latest reward value based on the real-time state difference feature vectors of each source station and the real-time state difference feature vector of the user.

[0019] According to the present invention, a CDN multi-origin site back-to-origin strategy autonomous update method is provided, wherein the origin site state difference feature extraction module and the user state difference feature extraction module are both long short-term memory neural networks; the attention layer module includes: a first fully connected attention layer, a second fully connected attention layer and a fully connected output layer, wherein the first fully connected attention layer is connected to the second fully connected attention layer, and the second fully connected attention layer is connected to the fully connected output layer.

[0020] According to the present invention, a CDN multi-origin site back-to-origin strategy autonomous update method is provided, wherein the back-to-origin strategy autonomous update model includes: a first input layer, a first feature extraction layer, a second input layer, a second feature extraction layer, a third input layer, a third feature extraction layer, a fourth input layer, a fourth feature extraction layer, a merging layer, a first fully connected layer, a second fully connected layer, and an output layer.

[0021] The first input layer is connected to the first feature extraction layer to input the first source station state vector to the first feature extraction layer, and the first feature extraction layer is used to extract the source station state feature vector of the first source station state vector.

[0022] The second input layer is connected to the second feature extraction layer to input the first user state vector into the second feature extraction layer, and the second feature extraction layer is used to extract the user state feature vector of the first user state vector.

[0023] The third input layer is connected to the third feature extraction layer and is used to input the node state vector into the third feature extraction layer. The third feature extraction layer is used to extract the node state feature vector of the node state vector.

[0024] The fourth input layer is connected to the fourth feature extraction layer and is used to input the source-return autonomous update action to the fourth feature extraction layer. The fourth feature extraction layer is used to extract the action feature vector of the source-return autonomous update action.

[0025] The merging layer is used to merge the source station state feature vector, user state feature vector, node state feature vector, and action feature vector to obtain a merged feature vector.

[0026] The first fully connected layer and the second fully connected layer are used to predict the value Q of the back-to-source autonomous update action based on the merged feature vector, and output the value Q through the output layer.

[0027] According to the present invention, a method for autonomously updating a CDN multi-origin site back-to-origin strategy is provided, wherein the training samples are constructed as follows.

[0028] Given an initial state, the corresponding state data is s. t and ct The s t Let c be the set of the first source station state sample vector and the first user state sample vector at the current time step t. t Given the node state sample vector at time step t, based on a greedy algorithm and s t Solve for the source-return self-update action sample a t .

[0029] Get the current reward r t And reach a new state, obtaining the new state data as s. t+1 .

[0030] Construct the training samples (s) at time t t , a t , c t , r t , s t+1 ), and store.

[0031] The present invention also provides a CDN multi-origin site back-to-origin strategy self-updating device, comprising the following units.

[0032] The data acquisition unit is used to acquire the first source station state vector of each source station, the first user state vector of the user application, and the node state vector of each CDN edge node in real time, and to normalize the first source station state vector, the first user state vector, and the node state vector respectively.

[0033] The source update action output unit is used to input the normalized first source station state vector, first user state vector and node state vector into the source strategy autonomous update model to obtain the source autonomous update action representing the load ratio of each source station output by the source strategy autonomous update model.

[0034] The origin update action distribution unit is used to distribute the origin self-update action to each CDN edge node. The origin self-update action is used to instruct the CDN edge node to perform origin recursion with the origin self-update action as the updated origin recursion strategy.

[0035] The source-following strategy autonomous update model is a deep reinforcement learning-based model, trained using (s,a,c,r,s') as training samples and the target Q-value Q(s,a,c) corresponding to (s,a,c,r,s') as the label. Here, s is the set of the first source station state sample vector and the first user state sample vector at the current time step, c is the node state sample vector at the current time step, r is the current reward value of the source-following strategy autonomous update model, s' is the set of the first source station state sample vector and the first user state sample vector at the next time step, and a is the source-following autonomous update action sample corresponding to s at the current time step.

[0036] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the CDN multi-origin site back-to-origin strategy autonomous update method as described above.

[0037] The present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the CDN multi-origin site back-to-origin strategy autonomous update method as described above.

[0038] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the CDN multi-origin site back-to-origin strategy autonomous update method as described above.

[0039] The CDN multi-origin site autonomous update method and apparatus provided by this invention, through an autonomous update model of the autonomous update of the origin strategy, outputs an autonomous update action representing the load ratio of each origin site based on the state vector of the first origin site, the state vector of the first user, and the state vector of the node. This enables CDN edge nodes to update their origin strategies based on the autonomous update action and perform origin recursion according to the updated strategy. Since the autonomous update model of the origin strategy obtains the load ratio of each origin site in real time based on the state of each origin site, user, and edge node, the CDN edge nodes perform origin recursion according to the updated load ratio of each origin site, achieving high accuracy and timeliness of origin recursion. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0041] Figure 1 This is a flowchart illustrating the CDN multi-origin site back-to-origin strategy self-update method provided by the present invention.

[0042] Figure 2 This is a schematic diagram of the reward neural network structure in the CDN multi-origin site back-to-origin strategy autonomous update method provided by the present invention.

[0043] Figure 3 This is a schematic diagram of the autonomous update model structure of the CDN multi-origin site back-to-origin strategy autonomous update method provided by the present invention.

[0044] Figure 4 This is a schematic diagram illustrating the specific application process of the CDN multi-origin site back-to-origin strategy self-update method provided by the present invention.

[0045] Figure 5 This is a schematic diagram of the CDN multi-source site back-to-source strategy autonomous update device provided by the present invention.

[0046] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0048] The CDN multi-origin site back-to-origin policy self-update method according to embodiments of the present invention, such as... Figure 1 As shown, the procedure includes the following steps S110 to S130.

[0049] Step S110: In real time, obtain the first source station state vector of each source station, the first user state vector of the user application (APP), and the node state vector of each CDN edge node, and normalize the first source station state vector, the first user state vector, and the node state vector respectively.

[0050] Specifically, each source station includes a primary source station and a backup source station. The state vector of the first source station can be characterized by the source station's key performance indicators (KPIs), including L dimensions such as request response latency, average throughput, average CPU utilization, data transfer volume, and request establishment success rate.

[0051] The first user state vector of a user application can be represented by the KPIs used by the application in the user terminal. These KPIs include F dimensions such as video opening latency, page click latency, image opening latency, video opening success rate, page opening success rate, and image opening success rate.

[0052] The node state vector of a CDN edge node is a multi-dimensional KPI of the CDN edge node, which includes G-dimensional attributes such as transmission network latency, bandwidth utilization, packet loss rate, data transmission volume, virtualization storage resource utilization, virtualization network resource utilization, virtualization computing resource utilization, number of error codes, and request success rate.

[0053] In this embodiment, the aforementioned state data is acquired in real time with a time granularity of 5 minutes, and each state data is normalized: normalization refers to scaling the data proportionally so that it falls into a small, specific range, that is, scaling the data to a given minimum and maximum value, usually between 0 and 1, so that the state data can be uniformly mapped to the range [0,1]. Normalization can be performed according to the following formula.

[0054] X_std = (X - X.min(axis=0)) / (X.max(axis=0) - X.min(axis=0)).

[0055] X_scaled = X_std * (max - min) + min.

[0056] The data attribute items in the first source station state vector, the first user state vector, and the node state vectors of each CDN edge node are normalized according to the above formula. The normalized data is beneficial to improving the convergence speed and accuracy of the model.

[0057] Step S120: Input the normalized first source station state vector, first user state vector and node state vector into the source return strategy autonomous update model to obtain the source return autonomous update action representing the load ratio of each source station output by the source return strategy autonomous update model.

[0058] The source-following strategy autonomous update model is a deep reinforcement learning-based model. It is trained using (s,a,c,r,s') as training samples and the target Q-value Q(s,a,c) corresponding to (s,a,c,r,s') as the label. Here, s represents the set of the first source station state sample vector and the first user state sample vector at the current time step, c represents the node state sample vector at the current time step, r represents the current reward value of the source-following strategy autonomous update model, s' represents the set of the first source station state sample vector and the first user state sample vector at the next time step, and a represents the source-following autonomous update action sample corresponding to s at the current time step. In this embodiment, the source-following autonomous update action is represented by the load ratio (0~100%) of each source station. That is, the source-following strategy autonomous update model can obtain the load ratio of each source station in real time based on the state of each source station, user, and edge node.

[0059] Step S130: The origin self-update action is sent to each CDN edge node. The origin self-update action instructs the CDN edge node to perform origin recursion using the updated origin recursion strategy. Specifically, the CDN edge node performs origin recursion according to the load ratio of each origin site. For example, if there are three origin sites A, B, and C, with load ratios of 10%, 30%, and 50% respectively, the CDN edge node divides the required resources into 8 parts, requests 5 parts of resources from origin site A, 3 parts of resources from origin site B, and 1 part of resources from origin site C.

[0060] In the CDN multi-origin site autonomous update method of this embodiment, the autonomous update model of the origin strategy outputs an autonomous update action representing the load ratio of each origin site based on the state vector of the first origin site, the state vector of the first user, and the state vector of the node. This enables the CDN edge nodes to update the origin strategy based on the autonomous update action and perform origin recursion according to the updated strategy. Since the autonomous update model of the origin strategy obtains the load ratio of each origin site in real time based on the state of each origin site, user, and edge node, the CDN edge nodes perform origin recursion according to the updated load ratio of each origin site, achieving high accuracy and timeliness of origin recursion.

[0061] In some embodiments, the CDN multi-origin site back-to-origin policy self-update method further includes the following steps.

[0062] Step 1: Obtain the second origin station state vector and the second user state vector of each origin station after the CDN edge node performs the origin pull. Specifically, after each CDN edge node performs the origin pull, the state of each origin station and the state of the user application will change. The second origin station state vector is the state vector of each origin station after the CDN edge node performs the origin pull, and the second user state vector is the state vector of the user application after the CDN edge node performs the origin pull.

[0063] Step 2: Calculate the source state difference between the second source station state vector and the first source station state vector, and calculate the user state difference between the second user state vector and the first user state vector.

[0064] Step 3: Input the source station state difference and user state difference into the reward neural network to obtain the latest reward value output by the reward neural network, and update the current reward value with the latest reward value. The reward neural network is trained using the time series of differences in the source station's historical state data and the corresponding time series of differences in the user's historical state data as samples, with the reward ground truth value corresponding to the time series of differences in the source station's historical state data as the label. The reward ground truth value is obtained through expert annotation. For example, for two adjacent historical state data (source station historical state data and user historical state data), after the corresponding historical back-to-source autonomous update action, if the changed historical state data of each source station and the user's historical state data both show performance improvement compared to their respective historical state data before the change, then... If the origin server status difference generally reflects an improvement in the KPIs of each origin server (e.g., an increase in average throughput), and the user status difference generally reflects an improvement in the user application KPIs (e.g., a reduction in video loading latency), then a larger reward truth value is set. For example, the reward truth value range can be defined as [1, 10], and a reward truth value of 10 or close to 10 can be set. If the KPIs of each origin server and the user application KPIs remain unchanged overall, then the reward truth value can be set to a value closer to the middle, such as 5. If the KPIs of each origin server and the user application KPIs decrease overall, then the reward truth value can be set to a value close to 1, such as 2. Therefore, the reward truth value can be obtained by marking the changes in the historical status data of the origin servers and the changes in the historical status data of the users before and after the historical origin server self-update action. In this embodiment, a reward neural network is used to predict the latest reward value based on the difference between the states of each source station and user before and after the source return, replacing the traditional manually designed reward function. The latest reward value is used to update the current reward value in the source return strategy self-update model, making the source return strategy self-update action predicted by the source return strategy self-update model more accurate and further improving the source return accuracy.

[0065] In some embodiments, such as Figure 2 As shown, the reward neural network is an attention-based reward neural network, including: a source station state difference feature extraction module, a user state difference feature extraction module, and an attention layer module. The source station state difference feature extraction module and the user state difference feature extraction module are both connected to the attention layer module.

[0066] The source station status difference feature extraction module is used to extract the real-time status difference feature vector of each source station based on the source station status difference of each source station.

[0067] The user state difference feature extraction module is used to extract the real-time state difference feature vector of the user based on the user state difference.

[0068] The attention layer module is used to predict and output the latest reward value based on the real-time state difference feature vectors of each source station and the real-time state difference feature vector of the user.

[0069] In this embodiment, both the source station state difference feature extraction module and the user state difference feature extraction module are Long Short-Term Memory (LSTM) neural networks. These modules utilize the long-term dependency information characteristics of the normalized state difference time series learned by the LSTM neural network to extract the real-time state difference feature vectors of each source station and the user. Specifically, the LSTM neural network consists of cascaded 128 LSTM neurons, with the activation function set to ReLU.

[0070] The attention layer module includes: a first fully connected attention layer, a second fully connected attention layer, and a fully connected output layer. The first fully connected attention layer is connected to the second fully connected attention layer, and the second fully connected attention layer is connected to the fully connected output layer.

[0071] Two fully connected attention layers (Dense attention) focus on the reward-related parts of the real-time state difference feature vectors of each source station and the real-time state difference feature vectors of the user, and assign different attention weights according to the correlation.

[0072] The attention weights are calculated based on the distribution of attention weights. i With the state difference feature vector s i+1 -s i The sum of the products of .

[0073] .

[0074] The fully connected output layer contains one Dense neuron, and the activation function is set to ReLU. p i As input, the output is the predicted reward value R(s) of the CDN multi-origin site back-to-origin strategy self-updating model. i+1 -s i ), s i Let represent the set of the first source station state vector and the first user state vector before the source retrieval. During model training, it can also be understood as the set of the first source station state sample vector and the first user state sample vector corresponding to the i-th time step. i+1 This represents the set of the second source station state vector and the second user state vector after back-to-source. During model training, it can also be understood as the set of the second source station state sample vector and the second user state sample vector corresponding to the (i+1)th time step, totaling... n Each time step.

[0075] During training, this reward neural network is trained for 1000 epochs (n=1000), with a batch size of 10. The Mean Squared Error (MSE) is chosen as the loss function and objective function (loss='mse') to calculate the predicted reward value. and reward truth value The training objective is to minimize the error between the two.

[0076] .

[0077] During training, the gradient descent optimization algorithm selects the Adam optimizer to improve the learning speed of traditional gradient descent (optimizer='adam'). The neural network uses gradient descent to find the optimal weight values ​​that minimize the objective function. As the number of training epochs increases, the training error gradually decreases, and the model gradually converges. After offline training is complete, the calculated neural network weights are exported, and the converged model becomes the reward neural network.

[0078] In some embodiments, the origin-following strategy self-update model described in this embodiment is constructed based on the deep reinforcement learning principle of DQN. This origin-following strategy self-update model takes as input historical multi-dimensional KPIs of each CDN origin server, historical multi-dimensional KPIs of end-user applications, and CDN edge node KPIs as conditions, along with the corresponding selected origin-following self-update action, and outputs the value Q(s,a,c) of this action selection. The Q value output by the origin-following strategy self-update model (equivalent to a critic network) is compared with the target Q value Q. target (s) i ,a i ,c i The algorithm compares and calculates the error, feeding the error signal back to the deep neural network to gradually improve model accuracy and select the autonomous update action that minimizes business impact and maximizes back-to-source efficiency. The error calculation (i.e., loss function) is as follows.

[0079] .

[0080] Q target (s i ,a i ,c i ) represents the target Q-value (i.e., the label), Q(s) i ,a i ,c i The Q-value of the model output is updated autonomously by the back-to-source strategy. a i This represents the source-following autonomous update action sample corresponding to the i-th time step.a This represents the source-based autonomous update action with the largest objective Q value among all source-based autonomous update actions corresponding to the (i+1)th time step. c i R(s) represents the node state sample vector of the CDN edge node at the i-th time step. i+1 -s i ) represents the reward value output by the reward neural network.

[0081] Specifically, such as Figure 3 As shown, the source-following strategy self-updating model includes: a first input layer, a first feature extraction layer, a second input layer, a second feature extraction layer, a third input layer, a third feature extraction layer, a fourth input layer, a fourth feature extraction layer, a merging layer, a first fully connected layer, a second fully connected layer, and an output layer.

[0082] The first input layer, connected to the first feature extraction layer, is used to input the first source station state vector into the first feature extraction layer. The first feature extraction layer is used to extract the source station state feature vector from the first source station state vector. Specifically, the first input layer receives the first source station state vector (s1) of each current CDN source station (including the primary source station and the backup source station). The first feature extraction layer consists of two Long Short-Term Memory (LSTM) layers, with 64 and 32 neurons respectively. The activation function for both layers is ReLU. The activation function introduces a nonlinear factor into the neurons, allowing the LSTM to arbitrarily approximate any nonlinear function. The ReLU activation function (The Rectified Linear Unit) is: relu = max(0,x).

[0083] The second input layer, connected to the second feature extraction layer, is used to input the first user state vector into the second feature extraction layer. The second feature extraction layer is used to extract the user state feature vector from the first user state vector. Specifically, the second input layer receives the first user state vector (s2) of the current user application. The second feature extraction layer consists of two long short-term memory (LSTM) layers, with 64 and 32 neurons respectively, and the activation function for both is ReLU.

[0084] The third input layer, connected to the third feature extraction layer, is used to input the node state vector into the third feature extraction layer. The third feature extraction layer is used to extract the node state feature vector from the node state vector. Specifically, the third input layer receives the node state vectors of each edge node of the current CDN as conditions (c). The third feature extraction layer consists of two long short-term memory (LSTM) layers, with 64 and 32 neurons respectively, and the activation function for both is ReLU.

[0085] The fourth input layer, connected to the fourth feature extraction layer, is used to input the source-return autonomous update action to the fourth feature extraction layer, which extracts the action feature vector of the source-return autonomous update action. Specifically, the fourth input layer receives the corresponding source-return autonomous update action (a). The fourth feature extraction layer consists of two fully connected layers (Dense), namely the third fully connected layer and the fourth fully connected layer, which have 32 and 16 neurons respectively, and both use ReLU activation function.

[0086] The merging layer is used to merge the source station state feature vector, user state feature vector, node state feature vector, and action feature vector to obtain a merged feature vector.

[0087] The first and second fully connected layers are used to predict the value Q of the origin-following autonomous update action based on the merged feature vector, and output the value Q through the output layer. The first and second fully connected layers (Dense) are configured with 32 and 16 neurons respectively, and the activation function is ReLU. The output layer consists of one fully connected neuron, outputting the Q value of the origin-following autonomous update action to be performed under the conditions of the CDN's origin station status, end-user application usage status, and CDN edge node status. The 'a' in this Q value represents the origin-following autonomous update action to be performed by each CDN edge node. It should be noted that during the training phase, the origin-following autonomous update actions input as samples to the fourth input layer can be obtained using the greedy algorithm described below. After training, the value Q will approach the target Q value. During the application phase of the origin-following strategy autonomous update model, only the real-time first origin station state vector, the real-time first user state vector, and the real-time node state vectors of each CDN edge node need to be input. The final output Q value will be obtained, and the 'a' in this Q value represents the origin-following autonomous update action to be taken by each CDN edge node.

[0088] In some embodiments, training data is randomly extracted from a memory that records the action, reward, and outcome (s, a, c, r, s') for each state and condition. The memory has a limited size; when it is full, the next data point overwrites the first. An experience replay buffer is used to store (s, a, c, r, s') for all stages in a replay memory. When training the neural network, updates are performed by randomly selecting mini-batches from this buffer, rather than using the most recent ones, thus addressing the issue of correlation between samples and significantly improving system stability.

[0089] In this embodiment, the training samples are constructed as follows.

[0090] Given an initial state, the corresponding state data is s. t and ct The s t Let c be the set of the first source station state sample vector and the first user state sample vector at the current time step t. t Given the node state sample vector at time step t, based on a greedy algorithm and s t Solve for the source-return self-update action sample a t To avoid limitations in action selection and to enrich data collection, a greedy algorithm (epsilon greedy) is introduced to select self-healing actions, that is, using epsilon (i.e., ...) as the basis for selection. ε Actions are randomly selected with a probability of 1 - epsilon, choosing the currently known optimal action. As learning progresses, the value of epsilon can become smaller and smaller, shifting the learning approach from extensive exploration to in-depth study.

[0091] .

[0092] Get the current reward r t And reach a new state, obtaining the new state data as s. t+1 .

[0093] Construct the training samples (s) at time t t , a t , c t , r t , s t+1 ), and store, specifically, in the playback cache.

[0094] In this embodiment, the process of autonomously updating the model training using the source retrieval strategy is as follows.

[0095] Initialize the Q-function with random weights, such that the objective Q-function Q0 is... target =Q, at each time step t in each round.

[0096] 1) Given an initial state s t Based on a greedy algorithm, a sample of source-based autonomous update action 'a' is given. t .

[0097] 2) Receive a reward r t And reached the new CDN origin status. t+1 .

[0098] 3) Transform the time t (s) t , a t , c t , r t , s t+1 Store it in the playback cache.

[0099] 4) Extract (s) from the playback bufferi , a i , c i , r i , s i+1 (), which is usually the number of a batch.

[0100] 5) Calculate the target Q value y=r i +maxQ target (s i+1 ,a,c i ).

[0101] 6) Update the parameters of the Q-function neural network in the model so that Q(s) i ,a i ,c i The closer it is to the target Q value y, the better.

[0102] 7) Assign the updated Q-function neural network weights to Q target =Q.

[0103] The model will be trained for 1000 epochs (epochs=1000), with a batch size of 32 and a replay buffer size of 50000. Mean Absolute Error (MSE) is chosen as the loss function and objective function (loss='mse'), and the Adam optimizer is selected to improve the learning speed of traditional gradient descent (optimizer='adam'). The neural network uses gradient descent to find the optimal weight values ​​that minimize the objective function. As the number of training epochs increases, the training error gradually decreases, and the model gradually converges. After offline training, the calculated neural network weights are exported.

[0104] The following is combined with Figure 4 The application process of the CDN multi-origin site back-to-origin strategy autonomous update method in the above embodiments is explained, specifically including the following steps.

[0105] Step 1.1: The CDN scheduling platform inputs multi-state data, namely the current real-time KPI (s1) of each origin station and the real-time KPI (s2) of the user, into the time series data preprocessing module.

[0106] Step 1.2: The CDN scheduling platform inputs the current CDN edge node KPI as condition (c) into the time series data preprocessing module.

[0107] Step 2: After normalizing and preprocessing the time series data, input it into the trained autonomous update model for CDN multi-origin site back-to-origin strategy.

[0108] Step 3: Model output returns to the source for autonomous update action (a), specifically the load ratio of each source station (0-100%).

[0109] Step 4: The CDN scheduling platform distributes the autonomous update action of the model output back to the source to each CDN edge node.

[0110] Step 5: Each CDN edge node performs an autonomous update action to retrieve the origin server, and retrieves the origin server according to the load ratio of each origin server's update.

[0111] Step 6: Each origin server sends its real-time KPIs to the nearest CDN edge node.

[0112] Step 7: The CDN edge node requests relevant end users to provide feedback on their current real-time application usage KPIs.

[0113] Step 8: Relevant end users report their current real-time application usage KPIs to the nearest CDN edge node.

[0114] Step 9: Each CDN edge node reports the updated real-time KPI status (s1') of each origin server and the real-time application usage KPI status (s2') of end users.

[0115] Step 10: The CDN scheduling platform inputs the real-time KPI status of each origin server and the real-time application usage KPI status of end users into the trained reward neural network R(s). i+1 -s i This replaces the traditional, manually designed reward function.

[0116] Step 11: Reward the neural network output to update the CDN multi-origin site back-to-origin strategy autonomous model.

[0117] The following describes the CDN multi-origin site back-to-origin strategy self-update device provided by the present invention. The CDN multi-origin site back-to-origin strategy self-update device described below and the CDN multi-origin site back-to-origin strategy self-update method described above can be referred to in correspondence.

[0118] The CDN multi-origin site back-to-origin policy autonomous update device of this invention, as described in this embodiment, is as follows: Figure 5 As shown, it includes the following functional units.

[0119] The data acquisition unit 510 is used to acquire the first source station state vector of each source station, the first user state vector of the user application, and the node state vector of each CDN edge node in real time, and to normalize the first source station state vector, the first user state vector, and the node state vector respectively.

[0120] The source update action output unit 520 is used to input the normalized first source station state vector, first user state vector and node state vector into the source strategy autonomous update model to obtain the source autonomous update action representing the load ratio of each source station output by the source strategy autonomous update model.

[0121] The origin update action distribution unit 530 is used to distribute the origin self-update action to each CDN edge node. The origin self-update action is used to instruct the CDN edge node to perform origin recursion with the updated origin recursion strategy based on the origin self-update action.

[0122] The source-following strategy autonomous update model is a deep reinforcement learning-based model, trained using (s,a,c,r,s') as training samples and the target Q-value Q(s,a,c) corresponding to (s,a,c,r,s') as the label. Here, s is the set of the first source station state sample vector and the first user state sample vector at the current time step, c is the node state sample vector at the current time step, r is the current reward value of the source-following strategy autonomous update model, s' is the set of the first source station state sample vector and the first user state sample vector at the next time step, and a is the source-following autonomous update action sample corresponding to s at the current time step.

[0123] Optionally, the CDN multi-origin site back-to-origin strategy self-update device further includes: a difference calculation unit and a reward value update unit.

[0124] The data acquisition unit is also used to acquire the second source station state vector of each source station and the second user state vector of the user application after the data is retrieved from the source.

[0125] The difference calculation unit is used to calculate the source station state difference between the second source station state vector and the first source station state vector, and to calculate the user state difference between the second user state vector and the first user state vector.

[0126] The reward value update unit is used to input the source station state difference and the user state difference into the reward neural network to obtain the latest reward value output by the reward neural network, and update the current reward value with the latest reward value.

[0127] The reward neural network is trained using the time series difference between the source station's historical state data and the time series difference between the corresponding user's historical state data as samples, and the reward truth value corresponding to the time series difference between the source station's historical state data as the label.

[0128] Optionally, the reward neural network includes: a source station state difference feature extraction module, a user state difference feature extraction module, and an attention layer module, wherein the source station state difference feature extraction module and the user state difference feature extraction module are both connected to the attention layer module.

[0129] The source station status difference feature extraction module is used to extract the real-time status difference feature vector of each source station based on the source station status difference of each source station.

[0130] The user state difference feature extraction module is used to extract the real-time state difference feature vector of the user based on the user state difference.

[0131] The attention layer module is used to predict and output the latest reward value based on the real-time state difference feature vectors of each source station and the real-time state difference feature vector of the user.

[0132] Optionally, both the source station state difference feature extraction module and the user state difference feature extraction module are long short-term memory neural networks; the attention layer module includes: a first fully connected attention layer, a second fully connected attention layer and a fully connected output layer, wherein the first fully connected attention layer is connected to the second fully connected attention layer, and the second fully connected attention layer is connected to the fully connected layer.

[0133] Optionally, the source-following strategy self-updating model includes: a first input layer, a first feature extraction layer, a second input layer, a second feature extraction layer, a third input layer, a third feature extraction layer, a fourth input layer, a fourth feature extraction layer, a merging layer, a first fully connected layer, a second fully connected layer, and an output layer.

[0134] The first input layer is connected to the first feature extraction layer to input the first source station state vector to the first feature extraction layer, and the first feature extraction layer is used to extract the source station state feature vector of the first source station state vector.

[0135] The second input layer is connected to the second feature extraction layer to input the first user state vector into the second feature extraction layer, and the second feature extraction layer is used to extract the user state feature vector of the first user state vector.

[0136] The third input layer is connected to the third feature extraction layer and is used to input the node state vector into the third feature extraction layer. The third feature extraction layer is used to extract the node state feature vector of the node state vector.

[0137] The fourth input layer is connected to the fourth feature extraction layer and is used to input the source-return autonomous update action to the fourth feature extraction layer. The fourth feature extraction layer is used to extract the action feature vector of the source-return autonomous update action.

[0138] The merging layer is used to merge the source station state feature vector, user state feature vector, node state feature vector, and action feature vector to obtain a merged feature vector.

[0139] The first fully connected layer and the second fully connected layer are used to predict the value Q of the back-to-source autonomous update action based on the merged feature vector, and output the value Q through the output layer.

[0140] Optionally, the training samples are constructed as follows: given an initial state, the corresponding state data is s. t and c t The s t Let c be the set of the first source station state sample vector and the first user state sample vector at the current time step t. t Given the node state sample vector at time step t, based on a greedy algorithm and s t Solve for the source-return self-update action sample a t Get the current reward r t And reach a new state, obtaining the new state data as s. t+1 Construct the training samples (s) at time t. t , a t ,c t , r t , s t+1 ), and store.

[0141] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640. The processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can invoke logical instructions stored in the memory 630 to execute a CDN multi-origin site back-to-origin policy autonomous update method, which includes the following steps.

[0142] The system acquires the first source station state vector, the first user state vector of the user application, and the node state vector of each CDN edge node in real time, and performs normalization processing on the first source station state vector, the first user state vector, and the node state vector respectively.

[0143] The normalized first source station state vector, first user state vector, and node state vector are input into the source return strategy autonomous update model to obtain the source return autonomous update action representing the load ratio of each source station output by the source return strategy autonomous update model.

[0144] The origin self-update action is sent to each CDN edge node. The origin self-update action is used to instruct the CDN edge node to perform origin recursion with the updated origin recursion strategy based on the origin self-update action.

[0145] The source-following strategy autonomous update model is a deep reinforcement learning-based model, trained using (s,a,c,r,s') as training samples and the target Q-value Q(s,a,c) corresponding to (s,a,c,r,s') as the label. Here, s is the set of the first source station state sample vector and the first user state sample vector at the current time step, c is the node state sample vector at the current time step, r is the current reward value of the source-following strategy autonomous update model, s' is the set of the first source station state sample vector and the first user state sample vector at the next time step, and a is the source-following autonomous update action sample corresponding to s at the current time step.

[0146] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0147] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the CDN multi-origin site back-to-origin strategy autonomous update method provided by the above methods. The method includes the following steps.

[0148] The system acquires the first source station state vector, the first user state vector of the user application, and the node state vector of each CDN edge node in real time, and performs normalization processing on the first source station state vector, the first user state vector, and the node state vector respectively.

[0149] The normalized first source station state vector, first user state vector, and node state vector are input into the source return strategy autonomous update model to obtain the source return autonomous update action representing the load ratio of each source station output by the source return strategy autonomous update model.

[0150] The origin self-update action is sent to each CDN edge node. The origin self-update action is used to instruct the CDN edge node to perform origin recursion with the updated origin recursion strategy based on the origin self-update action.

[0151] The source-following strategy autonomous update model is a deep reinforcement learning-based model, trained using (s,a,c,r,s') as training samples and the target Q-value Q(s,a,c) corresponding to (s,a,c,r,s') as the label. Here, s is the set of the first source station state sample vector and the first user state sample vector at the current time step, c is the node state sample vector at the current time step, r is the current reward value of the source-following strategy autonomous update model, s' is the set of the first source station state sample vector and the first user state sample vector at the next time step, and a is the source-following autonomous update action sample corresponding to s at the current time step.

[0152] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the CDN multi-origin site back-to-origin strategy autonomous update method provided by the above methods, the method comprising the following steps.

[0153] The system acquires the first source station state vector, the first user state vector of the user application, and the node state vector of each CDN edge node in real time, and performs normalization processing on the first source station state vector, the first user state vector, and the node state vector respectively.

[0154] The normalized first source station state vector, first user state vector, and node state vector are input into the source return strategy autonomous update model to obtain the source return autonomous update action representing the load ratio of each source station output by the source return strategy autonomous update model.

[0155] The origin self-update action is sent to each CDN edge node. The origin self-update action is used to instruct the CDN edge node to perform origin recursion with the updated origin recursion strategy based on the origin self-update action.

[0156] The source-following strategy autonomous update model is a deep reinforcement learning-based model, trained using (s,a,c,r,s') as training samples and the target Q-value Q(s,a,c) corresponding to (s,a,c,r,s') as the label. Here, s is the set of the first source station state sample vector and the first user state sample vector at the current time step, c is the node state sample vector at the current time step, r is the current reward value of the source-following strategy autonomous update model, s' is the set of the first source station state sample vector and the first user state sample vector at the next time step, and a is the source-following autonomous update action sample corresponding to s at the current time step.

[0157] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0158] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0159] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for autonomously updating a CDN multi-origin server origin return strategy, characterized in that, include: The system acquires the first source station state vector, the first user state vector of the user application, and the node state vector of each CDN edge node in real time, and performs normalization processing on the first source station state vector, the first user state vector, and the node state vector respectively. The normalized first source station state vector, first user state vector and node state vector are input into the source return strategy autonomous update model to obtain the source return autonomous update action representing the load ratio of each source station output by the source return strategy autonomous update model. The origin self-update action is sent to each CDN edge node. The origin self-update action is used to instruct the CDN edge node to perform origin recursion with the updated origin recursion strategy based on the origin self-update action. The source-following strategy autonomous update model is a deep reinforcement learning-based model, trained using (s,a,c,r,s') as training samples and the target Q-value Q(s,a,c) corresponding to (s,a,c,r,s') as the label. Here, s is the set of the first source station state sample vector and the first user state sample vector at the current time step, c is the node state sample vector at the current time step, r is the current reward value of the source-following strategy autonomous update model, s' is the set of the first source station state sample vector and the first user state sample vector at the next time step, and a is the source-following autonomous update action sample corresponding to s at the current time step.

2. The CDN multi-origin site back-to-origin strategy self-update method according to claim 1, characterized in that, Also includes: After the source is retrieved, the second source station state vector of each source station and the second user state vector of the user application are obtained. Calculate the source station state difference between the second source station state vector and the first source station state vector, and calculate the user state difference between the second user state vector and the first user state vector; The source station state difference and the user state difference are input into the reward neural network to obtain the latest reward value output by the reward neural network, and the current reward value is updated with the latest reward value; The reward neural network is trained using the time series difference between the source station's historical state data and the time series difference between the corresponding user's historical state data as samples, and the reward truth value corresponding to the time series difference between the source station's historical state data as the label.

3. The CDN multi-origin site back-to-origin strategy self-update method according to claim 2, characterized in that, The reward neural network includes: a source station state difference feature extraction module, a user state difference feature extraction module, and an attention layer module, wherein the source station state difference feature extraction module and the user state difference feature extraction module are both connected to the attention layer module; The source station status difference feature extraction module is used to extract the real-time status difference feature vector of each source station based on the source station status difference of each source station; The user state difference feature extraction module is used to extract a real-time user state difference feature vector based on the user state difference; The attention layer module is used to predict and output the latest reward value based on the real-time state difference feature vectors of each source station and the real-time state difference feature vector of the user.

4. The CDN multi-origin site back-to-origin strategy self-update method according to claim 3, characterized in that, Both the source station state difference feature extraction module and the user state difference feature extraction module are long short-term memory neural networks; The attention layer module includes: a first fully connected attention layer, a second fully connected attention layer, and a fully connected output layer. The first fully connected attention layer is connected to the second fully connected attention layer, and the second fully connected attention layer is connected to the fully connected output layer.

5. The CDN multi-origin site back-to-origin strategy self-update method according to claim 1, characterized in that, The source-following strategy self-updating model includes: a first input layer, a first feature extraction layer, a second input layer, a second feature extraction layer, a third input layer, a third feature extraction layer, a fourth input layer, a fourth feature extraction layer, a merging layer, a first fully connected layer, a second fully connected layer, and an output layer; The first input layer is connected to the first feature extraction layer and is used to input the first source station state vector to the first feature extraction layer. The first feature extraction layer is used to extract the source station state feature vector of the first source station state vector. The second input layer, connected to the second feature extraction layer, is used to input the first user state vector into the second feature extraction layer, and the second feature extraction layer is used to extract the user state feature vector from the first user state vector. The third input layer is connected to the third feature extraction layer and is used to input the node state vector to the third feature extraction layer. The third feature extraction layer is used to extract the node state feature vector of the node state vector. The fourth input layer is connected to the fourth feature extraction layer and is used to input the source-back-to-source autonomous update action to the fourth feature extraction layer. The fourth feature extraction layer is used to extract the action feature vector of the source-back-to-source autonomous update action. The merging layer is used to merge the source station state feature vector, user state feature vector, node state feature vector, and action feature vector to obtain a merged feature vector; The first fully connected layer and the second fully connected layer are used to predict the value Q of the back-to-source autonomous update action based on the merged feature vector, and output the value Q through the output layer.

6. The CDN multi-origin site back-to-origin strategy self-update method according to any one of claims 1 to 5, characterized in that, The training samples are constructed as follows: Given an initial state, the corresponding state data is s. t and c t The s t Let c be the set of the first source station state sample vector and the first user state sample vector at the current time step t. t Given the node state sample vector at time step t, based on a greedy algorithm and s t Solve for the source-return self-update action sample a t ; Get the current reward r t And reach a new state, obtaining the new state data as s. t+1 ; Construct the training samples (s) at time t t , a t , c t , r t , s t+1 ), and store.

7. A CDN multi-origin site origin return strategy self-update device, characterized in that, include: The data acquisition unit is used to acquire the first source station status vector of each source station, the first user status vector of the user application, and the node status vector of each CDN edge node in real time, and to normalize the first source station status vector, the first user status vector, and the node status vector respectively. The source update action output unit is used to input the normalized first source station state vector, first user state vector and node state vector into the source strategy autonomous update model to obtain the source autonomous update action representing the load ratio of each source station output by the source strategy autonomous update model. The origin update action distribution unit is used to distribute the origin self-update action to each CDN edge node. The origin self-update action is used to instruct the CDN edge node to perform origin recursion with the origin self-update action as the updated origin recursion strategy. The source-following strategy autonomous update model is a deep reinforcement learning-based model, trained using (s,a,c,r,s') as training samples and the target Q-value Q(s,a,c) corresponding to (s,a,c,r,s') as the label. Here, s is the set of the first source station state sample vector and the first user state sample vector at the current time step, c is the node state sample vector at the current time step, r is the current reward value of the source-following strategy autonomous update model, s' is the set of the first source station state sample vector and the first user state sample vector at the next time step, and a is the source-following autonomous update action sample corresponding to s at the current time step.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the CDN multi-origin site back-to-origin strategy autonomous update method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the CDN multi-origin site back-to-origin strategy autonomous update method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the CDN multi-origin site back-to-origin strategy autonomous update method as described in any one of claims 1 to 6.

Images