Base station energy-saving control method based on deep reinforcement learning and related equipment

Abstract

The invention provides a base station energy-saving control method based on deep reinforcement learning and related equipment, and the method comprises the steps: obtaining train flow data and rate demand data corresponding to a target base station; inputting the train flow data into the trained feature extraction network, and extracting flow features; determining energy-saving control schemes corresponding to a plurality of preset energy-saving strategies through a preset deep reinforcement learning network based on the flow characteristics and the rate demand data, and executing the energy-saving control schemes; wherein the preset deep reinforcement learning network minimizes the power consumption of the target base station under the condition that the preset deep reinforcement learning network meets the service quality requirement of the target base station. Through the technical scheme provided by the invention, the power consumption of the target base station can be minimized on the premise that the service quality requirement is met, and the energy-saving control schemes corresponding to the plurality of preset energy-saving strategies are determined, so that the rapid flow fluctuation of a railway scene can be difficult to effectively deal with.

Description

Technical Field

[0001] The present invention relates to the field of railway technology, and in particular to a base station energy-saving control method based on deep reinforcement learning and related equipment. Background Art

[0002] With the continued development of intelligent high-speed rail infrastructure, the demand for bandwidth for wireless data transmission has increased significantly. This, coupled with the high reliability requirements of high-speed rail control systems, has driven the application of 5G networks in railway scenarios. 5G-R networks integrate 5G technology into railway transportation systems and are widely used in key areas such as high-speed rail control, command, and dispatching communications. The implementation of 5G-R will further accelerate the pace of railway informatization and intelligentization, enhancing the railway system's independent innovation capabilities. However, the high energy consumption associated with the use of multiple antennas and dense deployment of base stations (BSs) in 5G-R networks has become a pressing challenge.

[0003] To reduce the energy consumption of 5G base stations, various energy-saving technologies have been proposed. However, the energy consumption of 5G-R systems in railway scenarios still faces unique challenges. First, traffic in railway scenarios is characterized by high bursts and long periods of idle time. This non-continuous traffic characteristic requires energy-saving solutions that can quickly adapt to drastic traffic fluctuations. Second, during the railway's nighttime maintenance period, the traffic load is almost zero, and this traffic pattern is significantly different from that of the public network. Therefore, directly applying energy-saving strategies from public networks to railway scenarios will result in significant energy efficiency losses. In addition, most of the current common energy-saving solutions only consider adopting a single energy-saving measure, but a single energy-saving strategy is difficult to effectively cope with the rapid traffic fluctuations in railway scenarios. Summary of the Invention

[0004] The present invention provides a base station energy-saving control method and related equipment based on deep reinforcement learning to solve the defects in the existing technology.

[0005] The present invention provides a base station energy-saving control method based on deep reinforcement learning, comprising: Obtain train flow data and speed demand data corresponding to the target base station; Inputting the train flow data into the trained feature extraction network to extract flow features; Based on the traffic characteristics and the rate requirement data, energy-saving control schemes corresponding to multiple preset energy-saving strategies are determined through a preset deep reinforcement learning network, and the energy-saving control schemes are executed; wherein, the preset deep reinforcement learning network minimizes the power consumption of the target base station while meeting the service quality requirements of the target base station.

[0006] According to a base station energy-saving control method based on deep reinforcement learning provided by the present invention, the multiple preset energy-saving strategies include a base station sleep strategy, an antenna selection strategy, and a symbol shutdown strategy, and the deep reinforcement learning network includes intelligent agents corresponding to the base station sleep strategy, the antenna selection strategy, and the symbol shutdown strategy, respectively; the energy-saving control schemes corresponding to the multiple preset energy-saving strategies are determined by the preset deep reinforcement learning network based on the traffic characteristics and the rate requirement data, including: Based on the traffic characteristics and the rate requirement data, determining the operating mode of the target base station through an agent corresponding to the base station sleep strategy; Based on the traffic characteristics and the rate requirement data, determining the number of antennas to be turned on for the target base station through an agent corresponding to the antenna selection strategy; Based on the traffic characteristics and the rate requirement data, the number of symbols turned on for the target base station is determined by an agent corresponding to the symbol turning off strategy.

[0007] According to a base station energy-saving control method based on deep reinforcement learning provided by the present invention, after determining the energy-saving control schemes corresponding to multiple preset energy-saving strategies through a preset deep reinforcement learning network based on the traffic characteristics and the rate demand data, the method further includes: Calculating the power consumption of the target base station based on the energy-saving control scheme; A reward value is calculated according to the power consumption, and the deep reinforcement learning network is updated according to the reward value.

[0008] According to a base station energy-saving control method based on deep reinforcement learning provided by the present invention, based on the traffic characteristics and the rate demand data, determining energy-saving control schemes corresponding to multiple preset energy-saving strategies through a preset deep reinforcement learning network, including: Based on the traffic characteristics and the rate demand data, calculating the action value function value of each action in the action space corresponding to each preset energy-saving strategy through the deep reinforcement learning network; Based on the action value function value, the optimal action corresponding to each of the preset energy-saving strategies is determined, and the energy-saving control scheme is obtained based on the optimal action.

[0009] According to a base station energy-saving control method based on deep reinforcement learning provided by the present invention, based on the traffic characteristics and the rate demand data, the action value function value of each action in the action space corresponding to each preset energy-saving strategy is calculated through the deep reinforcement learning network, including: Based on the rate demand data, filtering the action spaces corresponding to the respective preset energy-saving strategies through a rate constraint filter to obtain a first action space; Performing a filtering operation on the first action space through a power consumption constraint filter to obtain a second action space; Based on the traffic characteristics, the action value function value of each action in the second action space corresponding to each preset energy-saving strategy is calculated through the deep reinforcement learning network.

[0010] According to a base station energy-saving control method based on deep reinforcement learning provided by the present invention, after determining the energy-saving control schemes corresponding to multiple preset energy-saving strategies through a preset deep reinforcement learning network based on the traffic characteristics and the rate demand data, the method further includes: If, in the energy-saving control solution, the working mode of the target base station is the sleep mode, the coverage areas of the neighboring base stations of the target base station are adjusted by cell scaling technology so that the neighboring base stations provide services for the load of the target base station.

[0011] According to a base station energy-saving control method based on deep reinforcement learning provided by the present invention, the feature extraction network includes a spatial feature extraction module and a temporal feature extraction module; the train flow data is input into the trained feature extraction network to extract flow features, including: Inputting the train flow data into the spatial feature extraction module to extract spatial features; The spatial features are input into the time feature extraction module to extract the time features to obtain the flow features.

[0012] The present invention also provides a base station energy-saving control device based on deep reinforcement learning, comprising: an acquisition module configured to acquire train flow data and rate demand data corresponding to a target base station; an extraction module configured to input the train flow data into a trained feature extraction network to extract flow features; An execution module is configured to determine, based on the traffic characteristics and the rate requirement data, energy-saving control schemes corresponding to multiple preset energy-saving strategies through a preset deep reinforcement learning network, and execute the energy-saving control schemes; wherein, the preset deep reinforcement learning network minimizes the power consumption of the target base station while meeting the service quality requirements of the target base station.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, it implements any of the above-described base station energy-saving control methods based on deep reinforcement learning.

[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements any of the base station energy-saving control methods based on deep reinforcement learning as described above.

[0015] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned base station energy-saving control methods based on deep reinforcement learning.

[0016] The base station energy-saving control method and related equipment based on deep reinforcement learning provided by the present invention model the train flow data in the railway scenario as a Markov decision process problem because of its time correlation. The Markov decision process problem is solved by a preset deep reinforcement learning network. The technical solution provided by the present invention can minimize the power consumption of the target base station while meeting the service quality requirements, without relying on real-time flow information. The present invention can not only accurately estimate the long-term energy consumption of the target base station while ensuring the service quality constraints, but also use the spatiotemporal correlation of train flow data to get rid of the dependence on the train flow data at the next moment as a known input. In addition, the present invention determines the energy-saving control schemes corresponding to multiple preset energy-saving strategies, which can effectively cope with the rapid flow fluctuations in railway scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 It is a schematic diagram of an application scenario provided by an embodiment of the present invention.

[0019] Figure 2 It is a flow chart of the base station energy-saving control method based on deep reinforcement learning provided by the present invention.

[0020] Figure 3 Schematic diagram of the principle of the agent learning strategy in an embodiment of the present invention.

[0021] Figure 4 This is a diagram of the Conv-LSTM-DDQN network architecture in an embodiment of the present invention.

[0022] Figure 5 2 is an architecture diagram of an action space reduction network in an embodiment of the present invention.

[0023] Figure 6: This is the reward function convergence curve of the base station sleep strategy with and without the action space reduction network in an embodiment of the present invention.

[0024] Figure 7 : is the reward function convergence curve of the antenna selection strategy with and without the action space reduction network in an embodiment of the present invention.

[0025] Figure 8 A schematic diagram comparing the total power consumption of a base station within 24 hours when different algorithms are applied in an example of the present invention.

[0026] Figure 9 It is a structural diagram of the base station energy-saving control device based on deep reinforcement learning provided by the present invention.

[0027] Figure 10 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0028] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0029] The following combination Figures 1-10 The present invention describes a base station energy-saving control method based on deep reinforcement learning and related equipment.

[0030] Application scenarios of the present invention are as follows Figure 1 As shown in Figure 1. First, the base stations along the railway are modeled as a one-dimensional linear topology. The system consists of K base stations. The speed of each high-speed train is 180 km / h-220 km / h, and each train is connected to the nearest base station within the coverage area. The capacity of each base station and the traffic demand in the coverage area are respectively and Each base station is equipped with N remote radio units (RRUs), and each RRU is equipped with N A antennas. In an exemplary embodiment of the present invention, a single baseband unit (BBU) and dual RRUs are deployed. Symbol shutdown is also supported, with a maximum number of active symbols per subframe being NS. This invention combines three energy-saving techniques: base station sleep, antenna selection, and symbol shutdown, aiming to minimize power consumption while meeting Quality of Service (QoS) requirements.

[0031] Figure 2FIG is a flow chart of a base station energy saving control method based on deep reinforcement learning according to an exemplary embodiment. Figure 2 As shown, in an exemplary embodiment, the base station energy-saving control method based on deep reinforcement learning includes steps 210 to 230, which are described in detail as follows.

[0032] Step 210: Obtain train flow data and speed demand data corresponding to the target base station.

[0033] In the embodiment of the present invention, the train flow data has temporal and spatial correlation, and includes train flow sampling data of the target base station and its surrounding n neighboring base stations at the same time period every day. The speed demand data is the speed demand of the train.

[0034] Step 220: Input the train flow data into the trained feature extraction network to extract flow features.

[0035] In the embodiment of the present invention, a feature extraction network is pre-trained, and the feature extraction network extracts temporal features and spatial features from train flow data.

[0036] Step 230: Based on the traffic characteristics and the rate requirement data, determine the energy-saving control schemes corresponding to multiple preset energy-saving strategies through a preset deep reinforcement learning network, and execute the energy-saving control schemes; wherein, the preset deep reinforcement learning network minimizes the power consumption of the target base station while meeting the service quality requirements of the target base station.

[0037] In this embodiment of the present invention, because train flow data in railway scenarios is temporally correlated, it can be modeled as a Markov Decision Process (MDP) problem. Reinforcement learning (RL) is an effective tool for solving MDP problems. An intelligent agent learns to approach the long-term optimal strategy through continuous interaction with the environment. In short, the environment influences the agent's state. Based on this state, the agent selects the optimal action and applies it to the environment to achieve the optimization goal. The agent then selects the optimal action from the action space, selects an energy-saving control plan corresponding to multiple preset energy-saving strategies, and executes the energy-saving control plan.

[0038] The technical solution provided by the present invention minimizes the total power consumption of base stations along the railway while meeting QoS requirements, without relying on real-time traffic information. This method not only accurately estimates the long-term energy consumption of base stations while ensuring QoS constraints, but also leverages the spatiotemporal correlation of train traffic data to eliminate the reliance on next-moment train traffic data as a known input. Furthermore, the offline training feature significantly reduces online computational overhead, making this method both real-time and efficient, making it particularly suitable for energy-saving optimization of 5G-R networks in railway scenarios.

[0039] In an exemplary embodiment of the present invention, the multiple preset energy-saving strategies include a base station sleep strategy, an antenna selection strategy, and a symbol shutoff strategy; the deep reinforcement learning network includes agents corresponding to the base station sleep strategy, the antenna selection strategy, and the symbol shutoff strategy, respectively; and determining the energy-saving control schemes corresponding to the multiple preset energy-saving strategies through the preset deep reinforcement learning network based on the traffic characteristics and the rate requirement data includes: Based on the traffic characteristics and the rate requirement data, determining the operating mode of the target base station through an agent corresponding to the base station sleep strategy; Based on the traffic characteristics and the rate requirement data, determining the number of antennas to be turned on for the target base station through an agent corresponding to the antenna selection strategy; Based on the traffic characteristics and the rate requirement data, the number of symbols turned on for the target base station is determined by an agent corresponding to the symbol turning off strategy.

[0040] In the embodiment of the present invention, a combination of three energy-saving technologies is comprehensively considered, including base station sleep strategy, antenna selection strategy and symbol shutdown strategy. For the symbol shutdown strategy, the maximum number of activated symbols in a subframe is N. S The principle behind base station dormancy is that during periods of low traffic, some base stations can enter dormant mode or a low-power state to conserve energy. At the same time, macro base stations provide coverage to users, or small base stations extend coverage to ensure quality of service (QoS). Antenna selection technology allows for the disabling of RF channels corresponding to certain antennas. Symbol shutoff technology achieves energy savings by disabling specific symbols within a subframe.

[0041] Each energy-saving strategy is equipped with a corresponding intelligent agent to generate the strategy. Through the collaborative design of multiple energy-saving technologies, the goal is to minimize power consumption while meeting the quality of service requirements.

[0042] It's important to note that in the base station sleep strategy, when a base station enters sleep mode, it doesn't completely shut down the entire base station. Due to the complexity of starting and shutting down the BBU, the BBU remains powered on at all times. When the base station enters sleep mode, the RRU ceases service. This design ensures system responsiveness while achieving more efficient energy-saving optimization, making it particularly suitable for 5G-R network deployments in dynamic traffic and high-reliability railway scenarios.

[0043] The intelligent agents corresponding to the three energy-saving technologies may make decisions simultaneously or at different frequencies, and the present invention does not impose any restrictions on this.

[0044] When agents make decisions at different frequencies, for example, the decision interval for base station sleep decisions is set to 5 minutes. Within each 5-minute interval, the agent corresponding to the base station sleep decision selects the optimal operating mode and maintains the selected base station state. Subsequently, the agent corresponding to the antenna selection decision performs antenna selection at a 1-minute granularity, further determining the optimal number of antennas to be enabled based on the base station state. Subsequently, the agent corresponding to the symbol off decision performs symbol off selection at a 1-millisecond granularity, further determining the optimal number of symbols to be enabled based on the base station state.

[0045] In another embodiment of the present invention, the symbol selection strategy determines the number of symbols to be turned off, based on the combined results of two other agents, and is adjusted at a granularity of 1 millisecond. This multi-granularity decision-making mechanism fully combines long-term base station sleep, short-term antenna selection, and millisecond-level symbol turn-off optimization, achieving refined management of 5G-R network energy consumption in railway scenarios and significant energy savings.

[0046] In an exemplary embodiment of the present invention, after determining the energy-saving control schemes corresponding to the plurality of preset energy-saving strategies through a preset deep reinforcement learning network based on the traffic characteristics and the rate demand data, the method further includes: Calculating the power consumption of the target base station based on the energy-saving control scheme; A reward value is calculated according to the power consumption, and the deep reinforcement learning network is updated according to the reward value.

[0047] In the embodiment of the present invention, the power consumption of base station k at time t is expressed as: ; in, 、 、 and They represent the number of enabled antennas, the number of enabled symbols, the number of enabled RRUs and the cell coverage radius at time t respectively. 、 and The energy saving coefficient corresponding to each energy saving strategy. The power consumption of the base station also takes into account the basic power consumption of system components and their dynamic change characteristics. The fixed power consumption of the base station in working mode is , the dynamic power consumption is The baseline power consumption of other system components is P0. The power consumption of BBU It is a constant value regardless of traffic load. The power consumption of RRU is divided into static power consumption and dynamic power consumption . This value reflects the maximum dynamic power consumption of the RRU under maximum load conditions. Since the BBU power consumption is constant, it has a minimal impact on total power consumption; however, RRU power consumption is the primary factor affecting the overall power consumption of the base station. Furthermore, the change in cell coverage radius is described by the coefficient Rd(Ω), where Ω represents the size of the coverage radius. Dynamic changes in coverage radius directly affect the dynamic power consumption of the RRU, and thus the total power consumption of the base station. Under the symbol-off strategy, the number of active symbols is determined by the traffic load. As the traffic load decreases, the DTX function reduces the number of active symbols, thereby reducing power consumption.

[0048] In addition to the rated power consumption of the base station, this invention also specifically considers the additional power consumption caused by state switching. When a base station switches between different operating modes (for example, from sleep mode to non-sleep mode), it consumes additional energy. The impact of this state transition power consumption on the total power consumption in a practical system must be considered. The total power consumption can be expressed as: .

[0049] Based on the above description of the power consumption of the base station, the optimization problem of the present invention can be summarized as follows: In the present invention, the optimization objective function is shown in formula (a), which aims to minimize the total power consumption of the 5G-R system. The value of the total power consumption depends on 、 、 as well as The four decision variables are the number of antennas turned on, the number of symbols turned on, the number of RRUs turned on, and the base station working mode of base station k at time t. In order to ensure that the user quality of service (QoS) is not affected, a series of constraints need to be met. First, formula (b) indicates that the capacity provided by base station k must be greater than the traffic load of the working base station after redistribution, to ensure that when the base station enters sleep mode, its neighboring base stations can take over the load and meet user needs. In addition, in order to avoid the problem of insufficient cell coverage caused by multiple base stations entering sleep mode in succession, constraints are introduced. As shown in formula (c), the number of base stations that enter sleep mode in succession is limited to no more than two under any circumstances, thereby ensuring the stability of network coverage. At the same time, in order to ensure the reliability of communication at train stops, all base stations designated as stops must always remain active. This requirement is expressed by formula (d), where L is the set of stop station base stations. Finally, formulas (e), (f), and (g) define the value range of the decision variables to ensure 、 、 This approach is of practical significance and meets the needs of network operation. By optimizing decision variables and combining multiple constraints, this invention successfully minimizes power consumption while ensuring QoS stability and reliable network coverage. It is particularly suitable for 5G-R network energy-saving design in highly dynamic railway scenarios.

[0050] The state space at time t contains the key information required for strategy learning, including the rate demand vector R of K base stations and the base station sleep vector B. The state space can be expressed as The action space at time t is composed of the action vector of base station k Composition, written as Furthermore, the action vector of each base station is expressed as , including the number of symbols turned on, the number of antennas turned on, and the operating status of the base station. For networks that are suitable for a specific energy-saving strategy, only the actions related to that strategy need to be included in the action space. This design simplifies the decision space, allowing the agent to focus on decision variables related to the target strategy, thereby improving learning efficiency and optimization results.

[0051] The reward value is calculated through the set reward function, which consists of four main parts and is used to comprehensively consider energy saving goals, QoS, network operation security and other requirements. The reward function is expressed as: ; The first term of the reward function incentivizes energy-saving behavior by reducing total power consumption. That is, during the optimization process, the agent receives positive rewards by reducing the total power consumption of the base station, thereby guiding the optimization strategy towards a more energy-efficient approach. The second term of the reward function serves as a QoS constraint, ensuring that the throughput of base station k meets or exceeds the total traffic demand (including the initial traffic and the additional traffic due to cell scaling) through the following piecewise function: ; Among them, when x is less than zero (i.e., the base station throughput meets or exceeds the demand), no reward or penalty will be generated; when x is greater than zero (i.e., the base station throughput is lower than the demand), a penalty that increases exponentially with x is imposed, but the penalty value has a large upper limit to provide strict constraints. This design can effectively suppress situations where QoS cannot be met and encourage the agent to give priority to strategies that meet the demand. The third term of the reward function ensures that the key base stations are always in working condition through artificially defined priority control. Through the penalty coefficient Penalties are imposed on the sleep behavior of specific base stations to prevent key base stations (such as base stations at train stops) from entering sleep mode. The fourth term of the reward function limits the number of consecutive sleep base stations to ensure communication stability and train operation safety. The function is expressed as: ; The above formula shows that when the number of base stations that continuously enter the dormant state exceeds the maximum allowed, a penalty is imposed on the excess, thus limiting this occurrence. By defining this reward function, we can optimize energy consumption while strictly ensuring network QoS, critical base station activity, and communication security.

[0052] The resulting reward is used to update the action-value function (Q-value) to more accurately predict the value of each action in a given state. By continuously interacting with the environment and receiving rewards, the agent can gradually learn which actions in which states yield higher long-term rewards. By more accurately estimating Q-values, the agent selects the optimal action and updates the network parameters, thereby continuously optimizing the action selection strategy. The update of Q-values is determined by the Bellman equation: ; in, Indicates the Q value of the current Q network, the network parameters are θ, θ - are the parameters of the target Q network, which are periodically copied from the current Q network parameters θ. γ is a discount factor that weighs the importance of current rewards and future rewards, and α is the learning rate. represents the optimal action in the next state based on the current Q network selection.

[0053] This embodiment of the present invention employs an improved memory buffer replacement strategy. Rather than simply discarding experiences in the order they were received, the present invention uses a randomized approach to discard experiences, ensuring a more even distribution of traffic characteristics throughout the day. This strategy enhances the buffer's ability to capture diverse traffic patterns and improves the model's generalization across diverse scenarios.

[0054] In an exemplary embodiment of the present invention, determining energy-saving control schemes corresponding to multiple preset energy-saving strategies through a preset deep reinforcement learning network based on the traffic characteristics and the rate demand data includes: Based on the traffic characteristics and the rate demand data, calculating the action value function value of each action in the action space corresponding to each preset energy-saving strategy through the deep reinforcement learning network; Based on the action value function value, the optimal action corresponding to each of the preset energy-saving strategies is determined, and the energy-saving control scheme is obtained based on the optimal action.

[0055] In an embodiment of the present invention, based on traffic characteristics and rate demand data, the action value function value of each action in the action space corresponding to each preset energy-saving strategy is calculated through an intelligent agent. Based on the action value function value, the action corresponding to the largest action value function value is selected as the optimal action, and the optimal action corresponding to each preset energy-saving strategy is determined, thereby obtaining an energy-saving control plan and deploying it to the base station facility for execution.

[0056] In an exemplary embodiment of the present invention, the calculation of the action value function value of each action in the action space corresponding to each preset energy-saving strategy based on the traffic characteristics and the rate demand data through the deep reinforcement learning network includes: Based on the rate demand data, filtering the action spaces corresponding to the respective preset energy-saving strategies through a rate constraint filter to obtain a first action space; Performing a filtering operation on the first action space through a power consumption constraint filter to obtain a second action space; Based on the traffic characteristics, the action value function value of each action in the second action space corresponding to each preset energy-saving strategy is calculated through the deep reinforcement learning network.

[0057] In the embodiment of the present invention, the scale of the action space in the constructed network is very large and increases exponentially with the number of base stations. For example, in the antenna selection strategy, the scale of the action space reaches 8 K This poses a significant challenge to the agent's efficient exploration and finding of optimal solutions. Therefore, this paper designs a mechanism to effectively control the size of the action space. Through the Action Space Reduction Network (ASRN), invalid actions that do not meet QoS requirements or consume too much energy, such as those that enable too many base stations, antennas, or symbols, are eliminated. This reduces the size of the action space and significantly improves the agent's exploration efficiency. To identify these invalid actions, the ASRN designs two filters: 1) Rate constraint filter F RC . F RC Check whether the decision meets the user's rate requirements and eliminate infeasible decisions that violate rate constraints. Rate constraints are implemented by checking whether the following function is above a threshold, namely: ; represents the service demand within the coverage area of base station k at time t, It represents the cell throughput that base station k can provide at time t. When ≤ 0, the throughput provided by the base station meets the rate requirements of its coverage area and the decision is feasible; when When ≥0, the filter removes these invalid decisions. After removing all infeasible decisions from the complete action space, the first action space A that satisfies the rate constraint is obtained. R ,Right now: ; in, is a small positive value, such as 10 -5 , A is the complete action space.

[0058] 2) Power Constraint Filter. The power constraint filter checks the action space A R The system determines whether the actions in the decision result in excessive energy consumption and removes these actions. For example, if a decision turns on all base stations and antennas, this action consumes too much power and will be filtered out. The power consumption constraint is defined by the following conditions: ; Among them, P th is the maximum power consumption limit. In the present invention, P th Set it to 70% of the maximum power consumption of all base stations without energy-saving strategies. By filtering actions that do not meet the power consumption constraints, a new action space is obtained, namely the second action space A. D : ; In the second action space A D In the example, all actions satisfy the energy consumption and rate constraints. D The optimal action is selected based on the maximum Q value. Since a large number of infeasible actions are removed in the process of reducing the action space, the exploration efficiency of the intelligent agent is significantly improved. Finally, the selected actions are deployed to the base station facilities for execution, thereby achieving optimal allocation of network resources and effective reduction of power consumption. In this way, the present invention not only solves the problem of excessive action space, but also improves the efficiency of strategy learning, enabling the intelligent agent to quickly find energy-saving optimization solutions in complex 5G-R network scenarios. At the same time, thanks to the CLDN network's capture of the spatiotemporal correlation of traffic, the present invention does not need to rely on instantaneous traffic, but only needs to make decisions through historical traffic data.

[0059] In an exemplary embodiment of the present invention, after determining the energy-saving control schemes corresponding to the plurality of preset energy-saving strategies through a preset deep reinforcement learning network based on the traffic characteristics and the rate demand data, the method further includes: If, in the energy-saving control solution, the working mode of the target base station is the sleep mode, the coverage areas of the neighboring base stations of the target base station are adjusted by cell scaling technology so that the neighboring base stations provide services for the load of the target base station.

[0060] In the embodiment of the present invention, Figure 1 As shown in Figure 2, assuming that K base stations are deployed along the railway, their adjacency relationship can be represented by the base station adjacency matrix A, which can be written as: ; Among them, element a ij Indicates whether base station i and base station j are adjacent. Definition a ij =1, base station i is adjacent to base station j; a ij =0, indicating that the two base stations are not adjacent. The adjacent state of each base station is defined as a ii = 0. Since base stations along the railway are deployed in a linear topology, each base station only has an adjacency relationship with the adjacent base stations in front and behind it.

[0061] The present invention describes the base station state at time t by the base station sleep vector: ; in, Represents the state of base station k, when =1, indicating that the base station is not in sleep state; =0, indicating that the base station is in sleep state.

[0062] When a base station is dormant, cell scaling technology is used to allow adjacent base stations to take over the load of the dormant base station by expanding their coverage area, thereby achieving efficient energy saving while ensuring the QoS of new users. The mathematical model of coverage area expansion is described as: ; when a ij b j = 1, the coverage radius of the working base station (adjacent to the dormant base station j) extends to distance d, where d is the distance between the centers of base stations i and j. When base station i is surrounded by multiple adjacent dormant base stations, its coverage radius is the maximum distance from base station i to the farthest adjacent base station. On the contrary, when a ij b j = 0, it means that all the adjacent base stations around base station i are in working state and their coverage radius remains unchanged. i =0, it means that base station i itself is in a dormant state and its coverage radius is reduced to zero.

[0063] In an exemplary embodiment of the present invention, the feature extraction network includes a spatial feature extraction module and a temporal feature extraction module; inputting the train flow data into the trained feature extraction network to extract flow features includes: Inputting the train flow data into the spatial feature extraction module to extract spatial features; The spatial features are input into the time feature extraction module to extract the time features to obtain the flow features.

[0064] In this embodiment of the present invention, the spatial feature extraction module is based on the convolutional layer and is used to capture spatial features from train flow data. The temporal feature module is based on the LSTM layer and is used to capture temporal features and analyze the temporal changes and correlations in the feature sequence, that is, the dynamic patterns of train flow data in the temporal dimension.

[0065] In an exemplary embodiment of the present invention, the principle of the base station energy-saving control method based on deep reinforcement learning is as follows: Figure 3 As shown in the figure, the agent obtains rate demand data and train flow data from the environment, updates the state space through the proposed Conv-LSTM-DDQN (CLDN) framework, which can capture the spatiotemporal correlation of train flow and make decisions without the need for flow information at the next moment. The agent selects the optimal action from the action space, determines the number of symbols turned on, the number of antennas turned on, and the base station working mode; after applying the determined action to the environment, the agent state is updated. The agent calculates the reward value based on the long-term total power consumption of the entire network and adjusts the decision-making strategy to minimize the cumulative reward.

[0066] In the training phase, there are two layers of architecture. The agent learns and optimizes the strategy to minimize the long-term cumulative energy consumption through repeated interactions with the simulation environment. Figure 4 As shown, the agent responsible for the base station sleep strategy is trained using five-minute average traffic data. The input to the training network consists of sampled data from the target base station and its surrounding n base stations at the same time of day. The convolutional layer extracts spatial features from the input data and flattens the feature map into a shape suitable for LSTM processing. The data then passes through the LSTM layer, which captures temporal features and analyzes temporal changes and correlations within the feature sequence. This process enables the model to better learn complex spatiotemporal interactions. Finally, the fully connected layer maps the LSTM output to the DDQN framework.

[0067] The agent selects the optimal action based on its current state. The action is then applied to the simulation environment. The DDQN network then uses the reward function and the Bellman equation to more accurately estimate the Q value to select the optimal action and update the evaluation network parameters, thereby continuously optimizing the action selection strategy. The agent's state is updated and traffic data from the environment within the current five minutes is obtained. During this process, invalid actions that do not meet QoS requirements or have excessive energy consumption must be eliminated through the ASRN network. The above process is repeated to continuously update the learning strategy. The network parameters are updated, and the mean square error (MSE) is used as the loss function, i.e.: ; Among them, target is the target value, that is: .

[0068] After training is complete, the testing phase begins. During this phase, the average traffic data collected within the decision-making granularity time period is fed into the trained Conv-LSTM-DDQN model in the corresponding format. The model then predicts the optimal decision for the next time period. Finally, based on the decisions made by the agent corresponding to the base station sleep strategy, the agent corresponding to the antenna selection strategy is trained in a similar manner to the above steps. After obtaining the base station operating status and the number of antennas turned on at the corresponding time granularity, the base station's adaptive symbol shutdown function automatically determines the number of symbols turned on. Once the specific values for the three energy-saving strategies at the minimum time granularity are determined, the decision is deployed in the base station equipment room.

[0069] In the embodiment of the present invention, Figure 5 The flow diagram of the ASRN network is presented, depicting the key steps of action space reduction and its role in the reinforcement learning framework. ASRN gradually reduces the action space through rate constraint filters and power constraint filters, and ensures that the remaining actions meet the optimization goal. First, the agent starts from the complete action space A based on the current state and uses the rate constraint filter to eliminate all actions that cannot meet the user's rate requirements. Specifically, the filter calculates whether each action meets the rate constraint conditions, removes all infeasible actions, and thus obtains the action subset A that meets the rate constraint. R This stage ensures that the agent only explores actions that can meet QoS requirements, significantly reducing the possibility of invalid search. Next, ASRN performs a R Apply the power constraint filter to detect whether there is a maximum power consumption limit P th This step further reduces the action space, ensuring that the remaining actions not only meet the rate requirements but also have a reasonable power consumption level, thus obtaining the final action subset A. D After completing the action space reduction, the agent moves from A D The optimal action is selected based on the maximum Q value and deployed to the base station facility for execution. The efficiency of this process comes from the significant reduction of the action space, which enables the agent to find the optimal strategy more quickly while reducing computational overhead.

[0070] like Figure 6 As shown, Figure 6 The reward function convergence curves of base station dormancy with and without the action space reduction network (ASRN) are compared. Figure 6In the example, the reward curve for the base station sleep period gradually rises and eventually converges, demonstrating that the proposed method ensures convergence. After approximately 300 training iterations, the agent equipped with the ASRN successfully finds a near-optimal strategy. Furthermore, compared to the network without the ASRN, the network equipped with the ASRN exhibits a significant improvement in cumulative reward at convergence, further validating the effectiveness of the ASRN in improving performance.

[0071] Figure 7 The reward function convergence curve for the antenna selection strategy is presented, showing that the network equipped with ASRN is more stable during the convergence process and has significantly higher reward values. This improvement is attributed to the action space reduction function provided by ASRN, which can effectively remove suboptimal actions that consume too much energy or fail to meet capacity requirements, thereby improving exploration efficiency and reducing the risk of the agent being trapped in a local optimal solution.

[0072] Furthermore, the difference in performance between the agents corresponding to the antenna selection strategy and the base station sleep strategy with and without the ASRN can be attributed to the significantly larger action space of the antenna selection strategy compared to the base station sleep strategy. This large action space makes it more difficult to find near-optimal decisions without the support of the ASRN, further highlighting the key role of the ASRN in improving decision-making efficiency and optimizing performance in complex scenarios.

[0073] Figure 8 In an exemplary embodiment, 12 base stations are selected and the total power consumption of all base stations is counted for one minute every hour for a total of 24 hours. The simulation parameters are set as N = 2, N A is 4, N S The algorithm proposed in this invention is compared with the following four benchmark algorithms: (1) All BS on This scheme keeps all base stations in working state at all times, and only uses antenna selection and symbol shutdown as energy-saving strategies. Figure 8 In , the blue curve represents the performance of the algorithm.

[0074] (2) Sequential on / off This solution uses a geographically ordered base station switching strategy. Initially, all base stations are operational. Then, they are shut down one by one in geographical order. If system capacity is insufficient after a shutdown, the base station is reactivated, and the system continues with the next base station in geographical order. The goal is to find the minimum combination of operational base stations that meets capacity requirements.

[0075] (3) Greedy The algorithm initially starts with all base stations operating, but prioritizes shutting down base stations with the lowest energy consumption (sorted by energy consumption from low to high). Each time a base station is shut down, the system's total energy consumption and capacity requirements are recalculated to ensure that the remaining base stations still have enough capacity to meet traffic demand.

[0076] (4) Genetic Algorithm (GA) Genetic algorithm is used as a comparison benchmark to evaluate its performance in base station energy saving optimization.

[0077] Among them, the algorithm proposed in this invention does not provide instantaneous traffic information, only the traffic data for the first five time periods. Other algorithms all provide instantaneous traffic data. The results show that the proposed algorithm achieves a 17.66% reduction in total power consumption compared to the baseline algorithm with all base stations turned on. In addition, compared with the genetic algorithm that takes into account the power consumption of base station switching, the proposed algorithm reduces power consumption by 1.02%. At the same time, when compared with the energy-saving performance of other benchmark algorithms, it can be seen that the energy-saving effect of the method proposed in this invention is significantly better than that of other methods.

[0078] Table 1 below shows a comparison of energy saving percentages of different algorithms. Taking full base station activation as the benchmark algorithm, it can be seen that the energy saving percentage of the method proposed in the present invention is significantly better than that of other methods.

[0079] Of particular note, the proposed method does not require real-time traffic information, which is a necessary input for other benchmark algorithms. This feature not only simplifies the complexity of actual deployment but also further improves the algorithm's applicability and robustness in dynamic network environments.

[0080] Finally, according to the test, the genetic algorithm takes 36.11 seconds to make each decision, while the trained CLDN model only takes 4.8 milliseconds. The algorithm proposed in this invention greatly improves the decision-making speed and is conducive to the rapid deployment of base station energy-saving strategies.

[0081] The following describes the base station energy-saving control device based on deep reinforcement learning provided by the present invention. The base station energy-saving control device based on deep reinforcement learning described below and the base station energy-saving control method based on deep reinforcement learning described above can refer to each other. It should be noted that the device provided in the following embodiments and the method provided in the above embodiments belong to the same concept, and the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment and will not be repeated here.

[0082] In an exemplary embodiment of the present invention, see Figure 9 , Figure 9A base station energy-saving control device based on deep reinforcement learning is shown according to an exemplary embodiment, including the following modules.

[0083] An acquisition module 910 is configured to acquire train flow data and rate demand data corresponding to a target base station; An extraction module 920 is configured to input the train flow data into a trained feature extraction network to extract flow features; The execution module 930 is configured to determine, based on the traffic characteristics and the rate requirement data, energy-saving control schemes corresponding to multiple preset energy-saving strategies through a preset deep reinforcement learning network, and execute the energy-saving control schemes; wherein, the preset deep reinforcement learning network minimizes the power consumption of the target base station while meeting the service quality requirements of the target base station.

[0084] In an exemplary embodiment of the present invention, the multiple preset energy-saving strategies include a base station sleep strategy, an antenna selection strategy, and a symbol shutoff strategy; the deep reinforcement learning network includes agents corresponding to the base station sleep strategy, the antenna selection strategy, and the symbol shutoff strategy, respectively; and the execution module 930 includes: A first determination submodule is configured to determine the operating mode of the target base station through an agent corresponding to the base station sleep strategy based on the traffic characteristics and the rate requirement data; A second determination submodule is configured to determine the number of antennas to be turned on of the target base station through an agent corresponding to the antenna selection strategy based on the traffic characteristics and the rate requirement data; The third determination submodule is configured to determine the number of symbols turned on by the target base station through an agent corresponding to the symbol shut-off strategy based on the traffic characteristics and the rate requirement data.

[0085] In an exemplary embodiment of the present invention, the base station energy-saving control device based on deep reinforcement learning further includes: A first calculation module is configured to calculate the power consumption of the target base station based on the energy-saving control scheme; The second computing module is configured to calculate a reward value according to the power consumption and update the deep reinforcement learning network according to the reward value.

[0086] In an exemplary embodiment of the present invention, the execution module 930 includes: a calculation submodule configured to calculate, based on the traffic characteristics and the rate demand data, an action value function value of each action in the action space corresponding to each preset energy-saving strategy through the deep reinforcement learning network; The fourth determination submodule is configured to determine the optimal action corresponding to each of the preset energy-saving strategies based on the action-value function value, and obtain the energy-saving control scheme based on the optimal action.

[0087] In an exemplary embodiment of the present invention, the computing submodule includes: a first filtering unit configured to filter the action spaces corresponding to the preset energy-saving strategies through a rate constraint filter based on the rate demand data to obtain a first action space; a second filtering unit configured to filter the first action space through a power consumption constraint filter to obtain a second action space; Based on the traffic characteristics, the action value function value of each action in the second action space corresponding to each preset energy-saving strategy is calculated through the deep reinforcement learning network.

[0088] In an exemplary embodiment of the present invention, the base station energy-saving control device based on deep reinforcement learning further includes: The adjustment module is configured to adjust the coverage area of the neighboring base stations of the target base station through cell scaling technology if the working mode of the target base station is the sleep mode in the energy-saving control scheme, so that the neighboring base stations provide services for the load of the target base station.

[0089] In an exemplary embodiment of the present invention, the feature extraction network includes a spatial feature extraction module and a temporal feature extraction module; the extraction module 920 includes: a first extraction submodule, configured to input the train flow data into the spatial feature extraction module to extract spatial features; The second extraction submodule is configured to input the spatial feature into the time feature extraction module, extract the time feature, and obtain the flow feature.

[0090] Figure 10 An example of a physical structure diagram of an electronic device is shown below. Figure 10 As shown, the electronic device may include: a processor 1010, a communication interface 1020, a memory 1030, and a communication bus 1040, wherein the processor 1010, the communication interface 1020, and the memory 1030 communicate with each other via the communication bus 1040. The processor 1010 may call the logic instructions in the memory 1030 to execute a base station energy-saving control method based on deep reinforcement learning, the method including: obtaining train flow data and rate demand data corresponding to the target base station; Inputting the train flow data into the trained feature extraction network to extract flow features; Based on the traffic characteristics and the rate requirement data, energy-saving control schemes corresponding to multiple preset energy-saving strategies are determined through a preset deep reinforcement learning network, and the energy-saving control schemes are executed; wherein, the preset deep reinforcement learning network minimizes the power consumption of the target base station while meeting the service quality requirements of the target base station.

[0091] Furthermore, the logic instructions in the aforementioned memory 1030 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0092] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the base station energy-saving control method based on deep reinforcement learning provided by the above methods, the method including: obtaining train flow data and rate demand data corresponding to the target base station; Inputting the train flow data into the trained feature extraction network to extract flow features; Based on the traffic characteristics and the rate requirement data, energy-saving control schemes corresponding to multiple preset energy-saving strategies are determined through a preset deep reinforcement learning network, and the energy-saving control schemes are executed; wherein, the preset deep reinforcement learning network minimizes the power consumption of the target base station while meeting the service quality requirements of the target base station.

[0093] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the base station energy-saving control method based on deep reinforcement learning provided by the above methods, the method comprising: obtaining train flow data and rate demand data corresponding to a target base station; Inputting the train flow data into the trained feature extraction network to extract flow features; Based on the traffic characteristics and the rate requirement data, energy-saving control schemes corresponding to multiple preset energy-saving strategies are determined through a preset deep reinforcement learning network, and the energy-saving control schemes are executed; wherein, the preset deep reinforcement learning network minimizes the power consumption of the target base station while meeting the service quality requirements of the target base station.

[0094] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0095] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A base station energy-saving control method based on deep reinforcement learning, characterized in that: include: Obtain train flow data and speed demand data corresponding to the target base station; Inputting the train flow data into the trained feature extraction network to extract flow features; Based on the traffic characteristics and the rate requirement data, energy-saving control schemes corresponding to multiple preset energy-saving strategies are determined through a preset deep reinforcement learning network, and the energy-saving control schemes are executed; wherein, the preset deep reinforcement learning network minimizes the power consumption of the target base station while meeting the service quality requirements of the target base station.

2. The base station energy-saving control method based on deep reinforcement learning according to claim 1, characterized in that: The multiple preset energy-saving strategies include a base station sleep strategy, an antenna selection strategy, and a symbol shutoff strategy, and the deep reinforcement learning network includes intelligent agents corresponding to the base station sleep strategy, the antenna selection strategy, and the symbol shutoff strategy, respectively; The energy-saving control schemes corresponding to multiple preset energy-saving strategies are determined based on the traffic characteristics and the rate demand data through a preset deep reinforcement learning network, including: Based on the traffic characteristics and the rate requirement data, determining the operating mode of the target base station through an agent corresponding to the base station sleep strategy; Based on the traffic characteristics and the rate requirement data, determining the number of antennas to be turned on for the target base station through an agent corresponding to the antenna selection strategy; Based on the traffic characteristics and the rate requirement data, the number of symbols turned on for the target base station is determined by an agent corresponding to the symbol turning off strategy.

3. The base station energy-saving control method based on deep reinforcement learning according to claim 1, characterized in that: After determining the energy-saving control schemes corresponding to the plurality of preset energy-saving strategies through a preset deep reinforcement learning network based on the traffic characteristics and the rate demand data, the method further includes: Calculating the power consumption of the target base station based on the energy-saving control scheme; A reward value is calculated according to the power consumption, and the deep reinforcement learning network is updated according to the reward value.

4. The base station energy-saving control method based on deep reinforcement learning according to claim 1, characterized in that: The energy-saving control schemes corresponding to multiple preset energy-saving strategies are determined based on the traffic characteristics and the rate demand data through a preset deep reinforcement learning network, including: Based on the traffic characteristics and the rate demand data, calculating the action value function value of each action in the action space corresponding to each preset energy-saving strategy through the deep reinforcement learning network; Based on the action value function value, the optimal action corresponding to each of the preset energy-saving strategies is determined, and the energy-saving control scheme is obtained based on the optimal action.

5. The base station energy-saving control method based on deep reinforcement learning according to claim 4, characterized in that: The calculating, based on the traffic characteristics and the rate demand data, the action value function value of each action in the action space corresponding to each preset energy-saving strategy through the deep reinforcement learning network includes: Based on the rate demand data, filtering the action spaces corresponding to the respective preset energy-saving strategies through a rate constraint filter to obtain a first action space; Performing a filtering operation on the first action space through a power consumption constraint filter to obtain a second action space; Based on the traffic characteristics, the action value function value of each action in the second action space corresponding to each preset energy-saving strategy is calculated through the deep reinforcement learning network.

6. The base station energy-saving control method based on deep reinforcement learning according to claim 1, characterized in that: After determining the energy-saving control schemes corresponding to the plurality of preset energy-saving strategies through a preset deep reinforcement learning network based on the traffic characteristics and the rate demand data, the method further includes: If, in the energy-saving control solution, the working mode of the target base station is the sleep mode, the coverage areas of the neighboring base stations of the target base station are adjusted by cell scaling technology so that the neighboring base stations provide services for the load of the target base station.

7. The base station energy-saving control method based on deep reinforcement learning according to any one of claims 1 to 6, characterized in that: The feature extraction network includes a spatial feature extraction module and a temporal feature extraction module; Inputting the train flow data into the trained feature extraction network to extract flow features includes: Inputting the train flow data into the spatial feature extraction module to extract spatial features; The spatial features are input into the time feature extraction module to extract the time features to obtain the flow features.

8. A base station energy-saving control device based on deep reinforcement learning, characterized in that: include: an acquisition module configured to acquire train flow data and rate demand data corresponding to a target base station; an extraction module configured to input the train flow data into a trained feature extraction network to extract flow features; An execution module is configured to determine, based on the traffic characteristics and the rate requirement data, energy-saving control schemes corresponding to multiple preset energy-saving strategies through a preset deep reinforcement learning network, and execute the energy-saving control schemes; wherein, the preset deep reinforcement learning network minimizes the power consumption of the target base station while meeting the service quality requirements of the target base station.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, it implements the base station energy-saving control method based on deep reinforcement learning as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the base station energy-saving control method based on deep reinforcement learning as described in any one of claims 1 to 7 is implemented.

Images