Base station energy-saving decision-making method for performing memory backtracking on DT based on state machine transfer

Abstract

The invention discloses a base station energy-saving decision-making method for carrying out memory backtracking on DT based on state machine transfer, a state machine memory module with a backtracking function is designed to be used for a Decision Transformer to carry out long-term stable decision-making on base station energy saving, and the method comprises the following steps: recording past information such as states, actions and environment rewards, and arranging the information into a time sequence according to a specific sequence; accumulating the environment rewards in the trajectory, and calculating an accumulated expected return (return-to-go) in the trajectory; the method comprises the following steps: constructing a decision model, and taking a Decision Transform as a main body model; designing a state machine memory module and a state machine backtracking module; determining a loss function and training a decision model, obtaining a model meeting requirements as the decision model by adjusting a learning rate, an optimizer and regularization parameters, and storing the decision model; and inputting environment historical data, and outputting decision actions by the large model. According to the method, the problem that a traditional offline reinforcement learning method Decision Transform suddenly loses efficacy in a long-term decision that the reasoning track length exceeds the training sample track length is solved, and the stability of the energy-saving decision of the base station is improved.

Description

Technical Field

[0001] This invention belongs to the field of decision-making technology and relates to a base station energy-saving decision-making method based on state machine transition memory backtracking of DT. Background Technology

[0002] As global 5G network construction enters a phase of large-scale deployment, energy consumption has become a key bottleneck restricting the sustainable development of the industry. The power consumption of 5G base stations (BSs) is 3-4 times that of 4G base stations. Therefore, global operators urgently need energy-saving solutions. Recent work has highlighted the challenges faced by base station on-off switching in 5G networks. Related articles have analyzed the power consumption model of 5G base stations and proposed four strategies to effectively reduce energy consumption: symbol shutdown, channel shutdown, carrier shutdown, and deep sleep.

[0003] In deep learning, base station energy-saving methods mainly focus on traffic prediction. For example, optimizing base station energy-saving control based on predicted traffic. In reinforcement learning, base station energy-saving schemes are mainly modeled as Markov decision processes, using reinforcement learning to switch between sleep modes to achieve energy saving. However, traditional reinforcement learning algorithms (such as Q-learning) suffer from slow convergence speed and poor portability, especially in complex and diverse environments with different base station distributions and traffic densities. Furthermore, reinforcement learning for base station energy-saving strategies requires exploration of real-world environments, which can lead to intermittent paralysis of the regional wireless network and generate unacceptable trial-and-error costs.

[0004] Recently, some work has proposed using Decision Transformer (DT) to achieve base station energy saving. This method accumulates future rewards as "returns-to-go" features, serializes them with historical states and actions into long trajectories, and constructs an offline dataset for learning. Compared with deep reinforcement learning, this method can converge faster in new scenarios and avoids the trial-and-error costs of online training. However, DT has a drawback: the trajectory length in the training samples is finite, so the value of the returns-to-go feature will not increase indefinitely. However, the trajectory length in actual deployment is infinite, so the value of the returns-to-go feature will increase indefinitely until it exceeds the model's understanding range.

[0005] Therefore, it is necessary to design a better algorithm that can improve the problem of returns-to-go features gradually increasing until they exceed the model's understanding range, thereby achieving continuous and stable base station energy-saving decision performance. Summary of the Invention

[0006] The purpose of this invention is to provide a base station energy-saving decision-making method based on state machine transitions and memory backtracking of DT, which solves the problem that traditional DT methods cannot make stable base station energy-saving decisions in an infinite process.

[0007] To achieve the above objectives, this invention provides a base station energy-saving decision-making method based on state machine transition memory backtracking of DT, comprising the following steps:

[0008] S1: Record the state, actions, and environmental rewards of the base station's energy-saving strategy in past interactions with the environment, and arrange them in a predetermined order to form a time-series sample set;

[0009] S2: Calculate the cumulative expected return;

[0010] S3: Design a state machine memory module. This module inputs the reward sequence composed of environmental rewards into an LSTM layer to obtain memory feature embeddings. After processing through a predefined query vector and key-value vector, it passes through a fully connected layer and a sigmoid activation function to obtain... If the first discriminant label b1 of the distribution is 1, then the normalized result of the memory feature embedding and the cumulative expected reward corresponding to the first time step of the reward sequence are added to the model cache list. and In the middle, a memory recall point is formed;

[0011] S4: Design a state machine backtracking module that calculates the memory feature embedding and the current time step. The key-value divergence among multiple cache vectors in the list is calculated, and these divergences are combined into a single key-value vector. A uniform bias is applied to the key-value vector, its reciprocal is taken, and its variance is calculated. Finally, the vector is passed through a gating unit to form a... The second discriminant label b2 of the distribution is used to determine whether to backtrack on the cumulative expected return at the current moment;

[0012] S5: Construct a decision model with Decision Transformer as the main model. This decision model obtains the backtracking results of the cumulative expected report based on the state machine memory module and the state machine backtracking module, and uses the time sequence sample set embedded with the backtracking results of the cumulative expected report as input.

[0013] S6: Determine the loss function and train the decision model.

[0014] Preferably, S1: Record the state, actions, and environmental rewards of the base station's energy-saving strategy in past interactions with the environment, and arrange them in a predetermined order to form a time-series sample set;

[0015] S2: Calculate the cumulative expected return;

[0016] S3: Design a state machine memory module. This module inputs the reward sequence composed of environmental rewards into an LSTM layer to obtain memory feature embeddings. After processing through a predefined query vector and key-value vector, it passes through a fully connected layer and a sigmoid activation function to obtain... If the discriminant label b1 of the distribution is 1, then the normalized result of the memory feature embedding and the cumulative expected reward corresponding to the first time step of the reward sequence are added to the model cache list respectively. and In the middle, a memory recall point is formed;

[0017] S4: Design a state machine backtracking module that calculates the memory feature embedding and the current time step. The KV divergence between multiple cache vectors in the list is used to form a KV vector. A uniform bias is applied to the KV vector, the reciprocal is taken, and the variance is calculated. Finally, a discrimination label b2 is formed through a gating unit, and it is used to determine whether to backtrack on the cumulative expected return at the current time.

[0018] S5: Construct a decision model with Decision Transformer as the main model. This decision model obtains the backtracking results of the cumulative expected report based on the state machine memory module and the state machine backtracking module, and uses the time sequence sample set embedded with the backtracking results of the cumulative expected report as input.

[0019] S6: Determine the loss function and train the decision model.

[0020] Preferably, the format of the time series sequence in step S1 is:

[0021]

[0022] in, , , Representing the first Each base station at time The state, actions, and cumulative expected returns, Indicates the number of base stations. This indicates the trajectory length of the sample.

[0023] Preferably, the format of the time series sequence in step S1 is:

[0024]

[0025] in, , , Representing the first Each base station at time The state, actions, and cumulative expected returns, Indicates the number of base stations. This indicates the trajectory length of the sample.

[0026] Preferably, in step S4 The formula for the discriminant label b1 of the distribution is:

[0027]

[0028] in, Indicates memory feature embedding, Represents the query vector. Represents a key-value vector. This indicates the transpose operation. Represents matrix multiplication. This represents the convolution operation. This indicates a fully connected layer.

[0029] Preferably, in step S5, it is determined whether to backtrack on the accumulated expected return at the current time based on the discrimination label b2, and the formula is as follows:

[0030]

[0031]

[0032] i = t+1, t+2, ,L

[0033] in, This represents the backtracking result of the cumulative expected return at time t. This represents the backtracking result of the accumulated expected return at time i. This represents the cumulative expected return at time i. This represents the discriminant vector.

[0034] Preferably, the key-value vector is a one-dimensional convolutional layer, and the query vector is a fully connected layer.

[0035] Preferably, the loss function in step S6 is:

[0036]

[0037] in, These represent the initial state and action in the time-series sample set. Indicates time action The probability distribution, express The predicted probability distribution Indicates time Discrimination Label The binary probability distribution, yes The predicted binary probability distribution.

[0038] To achieve the above objectives, the present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and when the program is executed on the processor, it implements the steps in the base station energy-saving decision method based on state machine transition for memory backtracking of DT as described above.

[0039] To achieve the above objectives, the present invention also provides a storage medium storing a computer program, which, when executed by a processor, implements the steps in the base station energy-saving decision-making method based on state machine transition for memory backtracking of DT as described above.

[0040] This invention addresses the problem of traditional Data Detection (DT) models failing to make long-term stable base station energy-saving decisions due to the gradual increase of return-to-go features beyond the model's comprehension range. It optimizes this by changing the calculation method of return-to-go features, modeling the environment as a state machine transition process, and designing specific structures for memory backtracking point discrimination and memory point backtracking. This allows the model's return-to-go features to adaptively backtrack, preventing excessive numerical accumulation and maintaining long-term stability of base station energy-saving decisions. Compared to reinforcement learning, the decision model designed in this invention does not rely on online interaction during training, enabling rapid training and convergence. Compared to traditional DT, the decision model designed in this invention can execute consistently reliable base station energy-saving decisions in long-term testing environments, solving the problem that traditional DT becomes unreliable once the testing environment time exceeds the trajectory sample size. Attached Figure Description

[0041] Figure 1 This is an overall flowchart of the method of the present invention.

[0042] Figure 2 This invention applies the model of the present invention to the state machine modeling of the scenario in the embodiment.

[0043] Figure 3 This is a structural diagram of the decision-making model of the present invention.

[0044] Figure 4 This is a flowchart of the state machine memory module of the present invention.

[0045] Figure 5 This is a flowchart of the state machine backtracking module of the present invention.

[0046] Figure 6 This is a schematic diagram of the energy consumption results from the base station energy-saving decision-making experiment.

[0047] Figure 7 This is a schematic diagram of the call drop rate results from the base station energy-saving decision-making experiment. Detailed Implementation

[0048] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0049] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0050] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0051] In base station energy-saving scenarios, a base station energy-saving decision-making method based on state machine transitions and memory backtracking of the time decomposition process (DT) is proposed. The scenario modeling is as follows: Figure 2 As shown, the interaction between the base station's energy-saving environment and the model is an infinite decision-making process. Some disturbances cause the environment to deviate from its stable point. The model, through a finite number of decision steps, brings the environment back to a stable state, and then repeats this process. We model this scenario as a state machine model that follows a Markov process and use the method proposed above to train a long-term stable decision model. The overall training process includes the following steps, such as... Figure 1 As shown:

[0052] S1: Records the state of the base station's energy-saving strategy in past interactions with the environment. ,action Environmental rewards Information such as these are arranged and organized in a specific order to form a time-series sequence sample set. Among them:

[0053] state This includes feature dimensions such as the base station's current sleep mode, next sleep mode, wake-up latency, number of antennas, user association mode, base station location, total power consumption, transmit power, number of users served, user service latency, user data transmission rate, and the minimum data transmission rate required by users. Here, f represents the feature. Indicates the feature number.

[0054] action Base station sleep action , These represent base station active status and three sleep statuses; base station antenna switching action. , These represent the states of idle antenna groups being turned on, held on, and turned off, respectively; and user association method actions. , These represent allowing new users to join, rejecting new users to join, and disconnecting all users, respectively.

[0055] Environmental awards For real values, the calculation method is as follows:

[0056]

[0057] in, The total power consumption of all base stations at that moment. For the service quality of all users at that moment, both features can be reflected in the state at the next moment. Obtained from Constraint penalty weights for custom balance values.

[0058] The initial (i.e., un-state machine backtracking) returns-to-go features are obtained by calculating the cumulative reward for each step, using the following formula:

[0059]

[0060] in, This represents the trajectory length of the sample. It's important to note that each state machine backtracking changes the returns-to-go value at the current time step and for all subsequent time steps. However, during the inference phase, future returns-to-go values ​​are inherently unknown and can only be represented by adding the single-step reward of the next time step to the current returns-to-go value. Therefore, state machine backtracking during the inference phase only needs to change the current returns-to-go value, and all subsequent returns-to-go values ​​will naturally change accordingly.

[0061] The time series obtained by embedding returns-to-go is represented as follows:

[0062]

[0063] in, , , Representing the first Each base station at time The sequence includes the state, action, and returns-to-go features. It begins with a state and ends with a state. This indicates the total number of base stations.

[0064] S2: Construct a decision-making model, with Decision Transformer as the main model. , , Process them to the same dimensional shape using separate linear layers. ,),in, It is the dimension of the unified high-dimensional space to which each feature is mapped after processing. Then, the torch.nn.Embedding() function is used to encode positional information and add it to each character in the sequence.

[0065] S3: Design a state machine memory module, such as Figure 4 As shown. First, define a reward window. Process the single-step reward r within the window to generate memory feature embeddings. A query vector and a key-value vector are predefined. The query vector Q is a vector with a size equal to... The fully connected layer has a key-value vector K with a size much smaller than 1. A one-dimensional convolutional layer with zero padding at the edges ensures that the size of the convolution result is consistent with the desired size. Alignment. Memory feature embedding After processing the query vector and key-value vector, the result is obtained through a fully connected layer and a sigmoid activation function. The discriminant label b1 for the distribution is given by the following formula:

[0066]

[0067] in, It is memory feature embedding, , , , It is the kernel size. , This indicates the transpose operation. Represents matrix multiplication. This represents the convolution operation. It is a fully connected layer. yes Activation function.

[0068] If the discriminant label b1 is approximately 1, then the normalized result of the memory feature embedding and the cumulative expected reward of the first time step of the reward sequence are added to the model cache list. and Among them It is a list of vectors, each vector corresponding to the normalized result of the memory feature embedding; It is a list of values, each value corresponding to the cumulative expected return at the first moment of the reward sequence. Each vector in Each value in the array corresponds one-to-one, and a pair of combinations is considered a memory backtracking point.

[0069] S4: Design a state machine backtracking module, such as Figure 5 As shown. This module calculates the memory feature embedding. With model cache list The KV divergence between each memory backtracking point is combined to form a discriminant vector. , This represents the number of memory backtracking points at the current moment. A learnable positive bias is applied to the discrimination vector, and its reciprocal is taken. Then, the variance of the entire vector is calculated and output through a gating unit. The discriminant label b2 of the distribution. b2 affects the memory backtracking at this moment and the enabling of the state machine memory module, as shown in the formula:

[0070]

[0071]

[0072] i = t+1, t+2, ,L

[0073] in This represents the backtracking result at time treturns-to-go. This represents the backtracking result at time ireturns-to-go. This represents the returns-to-go at time i, where the backtracking at time t affects all subsequent time steps. The calculation.

[0074] S5: Determine the loss function and train the base station energy-saving decision model. Adjust the learning rate, optimizer, and regularization parameters to obtain a model that meets the requirements, which is then saved as the final model. The loss function is set as a composite loss function, with the specific formula as follows:

[0075]

[0076] in, These represent the initial state and action recorded in the sample trajectory. It is a sample trajectory record Moment of action The probability distribution, It is a prediction The probability distribution. It is a sample trajectory record Whether a time is a backtracking point is a distinguishing label. The binary probability distribution, It is a prediction The binary probability distribution. It is used to fit decision actions The cross-entropy function, It is used to fit and discriminate labels. The cross-entropy function.

[0077] S6: Test the final model in an online base station energy-saving environment to evaluate the model's decision-making performance and further fine-tune it online.

[0078] Figure 3 The overall structure of the model is shown. The decision model's overall structure is as follows: the reward sequence is input into an LSTM layer to obtain a memory feature embedding. This memory feature embedding is first processed by a state machine backtracking module, outputting the processed intermediate result and the discriminant label b2. The discriminant label b2 is connected to the enable terminal of the state machine memory module, and the intermediate result is connected to the input terminal of the state machine memory module. Finally, the output is a self-backtracking return-to-go feature. The returns-to-go feature It is then integrated with the time series data into the Decision Transformer model.

[0079] Figure 6 and Figure 7 Demonstrated as Figure 2 The results of the base station energy-saving decision-making experiment in the illustrated scenario clearly show that the proposed method (ours) remains effective even when the traditional DT method collapses due to long-term continuous decision-making. Furthermore, the figure demonstrates that the proposed method also exhibits superior energy-saving performance compared to the traditional reinforcement learning method A2C.

[0080] The present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and when the program is executed on the processor, it implements the steps in the base station energy-saving decision-making method based on DT model and state machine transition backtracking as described above.

[0081] The present invention also provides a storage medium storing a computer program, which, when executed by a processor, implements the steps in the base station energy-saving decision-making method based on the DT model and state machine transitions for backtracking as described above.

[0082] It should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This way of describing the specification is only for clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0083] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the modules described above can be referred to the corresponding process in the aforementioned method implementation, and will not be repeated here.

[0084] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0085] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or in a combination of hardware and software functional modules.

[0086] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer system (which may be a personal computer, server, or network system, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A base station energy-saving decision-making method based on DT model and state machine transition backtracking, characterized in that: Includes the following steps: S1: Record the state, actions, and environmental rewards of the base station's energy-saving strategy in past interactions with the environment, and arrange them in a predetermined order to form a time-series sample set; S2: Calculate the cumulative expected return; S3: Design a state machine memory module. This module inputs the reward sequence composed of environmental rewards into an LSTM layer to obtain memory feature embeddings. After processing through a predefined query vector and key-value vector, it passes through a fully connected layer and a sigmoid activation function to obtain... If the first discriminant label b1 of the distribution is 1, then the normalized result of the memory feature embedding and the cumulative expected reward corresponding to the first time step of the reward sequence are added to the model cache list respectively. and In the middle, a memory recall point is formed; S4: Design a state machine backtracking module that calculates the memory feature embedding and the current time step. The key-value divergence among multiple cache vectors in the list is calculated, and these divergences are combined into a single key-value vector. A uniform bias is applied to the key-value vector, its reciprocal is taken, and its variance is calculated. Finally, the vector is passed through a gating unit to form a... The second discriminant label b2 of the distribution is used to determine whether to backtrack on the cumulative expected return at the current moment; S5: Construct a decision model with Decision Transformer as the main model. This decision model obtains the backtracking results of the cumulative expected report based on the state machine memory module and the state machine backtracking module, and uses the time sequence sample set embedded with the backtracking results of the cumulative expected report as input. S6: Determine the loss function and train the decision model; S7: Based on the trained decision model, long-term decision-making for base stations is achieved.

2. The method according to claim 1, characterized in that: In step S1: The status includes the base station's current sleep mode, next sleep mode, wake-up delay, number of antennas, user association mode, base station location, total power consumption, transmit power, number of users served, user service delay, user data transmission rate, and user minimum required data transmission rate; Actions include base station sleep mode, base station antenna on / off status, and user association method; Environmental awards ,in The total power consumption of all base stations at the current moment. For the quality of service for all users at the current moment, To balance the constraint penalty weights of numerical values.

3. The method according to claim 1, characterized in that: The format of the time series sequence in step S1 is as follows: ， in, , , Representing the first Each base station at time The state, actions, and cumulative expected returns, Indicates the number of base stations. This indicates the trajectory length of the sample.

4. The method according to claim 1, characterized in that: In the decision model of step S3, the state, action, and cumulative expected return are processed into the same dimensional shape using independent linear layers.

5. The method according to claim 1, characterized in that: In step S4 The formula for the first discriminant label of the distribution is: ， in, Indicates memory feature embedding, Represents the query vector. Represents a key-value vector. This indicates the transpose operation. Represents matrix multiplication. This represents the convolution operation. This indicates a fully connected layer.

6. The method according to claim 1, characterized in that: In step S5, it is determined whether to backtrack on the cumulative expected return at the current time based on the second discriminant label, and the formula is as follows: ， ， , i=t+1,t+2, ,L, in, This represents the backtracking result of the cumulative expected return at time t. This represents the backtracking result of the accumulated expected return at time i. This represents the cumulative expected return at time i. This represents the discriminant vector.

7. The method according to claim 1, characterized in that: The key-value vector is a one-dimensional convolutional layer, and the query vector is a fully connected layer.

8. The method according to claim 1, characterized in that: The loss function in step S6 is: ， in, These represent the initial state and action in the time-series sample set. Indicates time action The probability distribution, express The predicted probability distribution Indicates time First discriminant label The binary probability distribution, yes The predicted binary probability distribution.

9. An electronic device, comprising a memory and a processor, characterized in that, The memory stores a computer program that can run on the processor, and when the program is executed on the processor, it implements the steps of the method as described in any one of claims 1-8.

10. A storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-8.

Images