Method for selecting reinforcement learning algorithm and related device

Abstract

The embodiment of the invention provides a reinforcement learning algorithm selection method and a related device, and the method comprises the steps: obtaining the sampling capability information of a first device and / or a second device, and enabling the sampling capability information to be used for representing the information of the supporting capability of a sampling process, the sampling process is a process in which the first device communicates with the second device to obtain sampling content; the type of a reinforcement learning algorithm is determined according to the sampling capability information, the type of the reinforcement learning algorithm comprises a different strategy type or a same strategy type, the reinforcement learning algorithm defines a sampling process, and the reinforcement learning algorithm is used for realizing a communication function. According to the embodiment of the invention, the working stability of the communication system can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of wireless communication, and particularly relates to a selection method of a reinforcement learning algorithm and related apparatus. BACKGROUND

[0002] Reinforcement learning is an important machine learning method. Reinforcement learning defines two types of roles, i.e., an agent and an environment. In the reinforcement learning process, the agent interacts with the environment, i.e., the agent observes a state of the environment, then selects an action according to a corresponding policy and executes the action, the environment gives a reward according to the action of the agent, and enters a new state. Then, the agent adjusts the policy according to the reward of the environment to obtain the highest cumulative reward.

[0003] With the application of reinforcement learning in a communication system, the communication system can implement communication functions such as access control, link adaptation, power control, interference coordination, load balancing, and mobility management. However, the interaction process (also referred to as a sampling process) between the agent and the environment can affect the stability of the communication system when working. Therefore, how to improve the stability of the communication system when working is a technical problem that needs to be solved by those skilled in the art. SUMMARY

[0004] Embodiments of the present application provide a selection method of a reinforcement learning algorithm and related apparatus, which can improve the stability of the communication system when working.

[0005] In a first aspect, embodiments of the present application provide a selection method of a reinforcement learning algorithm, the method comprising:

[0006] obtaining sampling capability information of a first device and / or a second device, wherein the sampling capability information is information representing a support capability of a sampling process, and the sampling process is a process in which the first device and the second device communicate to obtain sampling content;

[0007] determining a type of the reinforcement learning algorithm according to the sampling capability information, wherein the type of the reinforcement learning algorithm includes a heterogeneous policy type or a homogeneous policy type, the reinforcement learning algorithm defines the sampling process, and the reinforcement learning algorithm is used to implement a communication function.

[0008] Considering that different types of reinforcement learning algorithms correspond to different sampling characteristics (i.e., a same-policy algorithm has more sampling times and less storage requirements, and a different-policy algorithm has less sampling times and more storage requirements), the above method can obtain the sampling capability information of at least one of the first device and / or the second device before the sampling process, and select a type of reinforcement learning algorithm that matches the support capability of the device for the sampling process according to the sampling capability information, which is beneficial to the first device to execute the reinforcement learning algorithm of the type, thereby reducing the impact of the sampling process on the stability of the communication system during operation, and further improving the stability of the communication system during operation.

[0009] In an optional implementation, the sampling content includes an action parameter, a state parameter, and a reward parameter, the action parameter is used to represent a feature of a communication operation performed by the first device on the second device in the sampling process, the state parameter is used to represent a loss of the communication operation, and the reward parameter is used to represent a network performance corresponding to the communication operation.

[0010] In another optional implementation, the sampling capability information includes one or more of a sampling frequency, a sampling energy consumption, a storage capacity, an air interface load, an interference degree, and a transmission requirement supported by the device.

[0011] In the above method, the support capability of the device for the sampling process represented by the sampling capability information includes a support capability of the device for the number of sampling times and a support capability of the device for a storage capacity required by the sampling, and because different types of reinforcement learning algorithms differ in the number of sampling times and storage requirements, the above method can more accurately determine a type of reinforcement learning algorithm that matches the support capability of the device for the sampling process, and further improve the stability of the communication system during operation.

[0012] In yet another optional implementation, the communication function is one or more of link adaptation, power control, interference coordination, load balancing, access control, and mobility management.

[0013] In yet another optional implementation, the communication function is power control, the communication operation is transmitting a communication signal with a first power, the action parameter is the first power, the state parameter is a path loss between the first device and the second device, and the reward parameter is a signal-to-interference-and-noise ratio of the communication signal.

[0014] In yet another optional implementation, the communication function is link adaptation, the communication operation is transmitting a communication signal with a first modulation and coding scheme, the action parameter is the first modulation and coding scheme, the state parameter includes channel state information (CSI) between the first device and the second device and a historical modulation and coding scheme, and the reward parameter is a transport block size of the communication signal.

[0015] In yet another optional implementation, the method is applied to the first device, and the obtaining the sampling capability of the first device and / or the second device comprises:

[0016] receiving the sampling capability information of the second device.

[0017] In yet another optional implementation, the method further comprises:

[0018] sending a first request message to the second device, wherein the first request message is used to request the sampling capability information.

[0019] In yet another optional implementation, the determining the type of the reinforcement learning algorithm according to the sampling capability information comprises:

[0020] determining the type of the reinforcement learning algorithm according to the sampling capability information of the first device and the second device.

[0021] In yet another optional implementation, the method further comprises:

[0022] receiving a first indication message, wherein the first indication message is used to indicate the use of the reinforcement learning algorithm.

[0023] In yet another optional implementation, the method further comprises:

[0024] selecting the reinforcement learning algorithm according to the type of the reinforcement learning algorithm, and executing the reinforcement learning algorithm.

[0025] In yet another optional implementation, the method further comprises:

[0026] obtaining, by the second device, sampling information of a third device.

[0027] In yet another optional implementation, the determining the type of the reinforcement learning algorithm according to the sampling capability information comprises:

[0028] determining the type of the reinforcement learning algorithm according to the sampling capability information of the first device, the second device and the third device.

[0029] In yet another optional implementation:

[0030] the first device is a base station, and the second device is a user equipment;

[0031] or, the first device is a user equipment, and the second device is a base station;

[0032] or, the first device and the second device are two different base stations;

[0033] or, the first device is a radio access network intelligent control (RIC) device, the second device is a base station, and the third device is a user equipment.

[0034] The method can be applied to various scenarios and various communication systems, and is widely applicable.

[0035] In yet another optional implementation, the method is applied to the second device, and obtaining the sampling capability information comprises:

[0036] Receiving the sampling capability information of the first device.

[0037] In yet another optional implementation, determining the type of the reinforcement learning algorithm according to the sampling capability information comprises:

[0038] Determining the type of the reinforcement learning algorithm according to the sampling capability information of the first device and the sampling capability information of the second device.

[0039] In yet another optional implementation, the method further comprises:

[0040] Sending the type information to the first device, wherein the type information comprises the type of the reinforcement learning algorithm.

[0041] In a second aspect, an embodiment of the present application provides a method for selecting a reinforcement learning algorithm, applied to a first device, the method comprising:

[0042] Receiving type information sent by a second device, wherein the type information comprises the type of the reinforcement learning algorithm, the type of the reinforcement learning algorithm comprises a heterogeneous policy type or a homogeneous policy type, the reinforcement learning algorithm is used to implement a communication function, the reinforcement learning algorithm defines a sampling process, the sampling process is a process in which the first device and the second device communicate to obtain sampling content, the type information is information determined according to sampling capability information of the first device and / or the second device, and the sampling capability information is information used to represent a support capability for the sampling process;

[0043] Selecting the reinforcement learning algorithm according to the type of the reinforcement learning algorithm, and executing the reinforcement learning algorithm.

[0044] Considering that different types of reinforcement learning algorithms correspond to different sampling characteristics (i.e., the homogeneous policy algorithm has more sampling times and smaller storage requirements, and the heterogeneous policy algorithm has fewer sampling times and larger storage requirements), the type of the reinforcement learning algorithm executed by the first device in the above method is an algorithm type that matches the support capability of the first device and / or the second device for the sampling process, and therefore, the above method can reduce the impact of the sampling process on the stability of the communication system during operation, thereby improving the stability of the communication system during operation.

[0045] In an optional implementation, the sampling content comprises an action parameter, a state parameter, and a reward parameter, the action parameter is used to represent a feature of a communication operation performed by the first device on the second device in the sampling process, the state parameter is used to represent a loss of the communication operation, and the reward parameter is used to represent a network performance corresponding to the communication operation.

[0046] In an optional implementation, the sampling capability information includes one or more of a sampling frequency supported by the device, a sampling energy consumption, a storage capacity, an air interface load, an interference level, and a transmission requirement.

[0047] In the method, the support capability of the device for the sampling process represented by the sampling capability information includes a support capability of the device for a sampling number and a support capability of the device for a storage capacity required for sampling. Since different types of reinforcement learning algorithms are different in sampling number and storage requirement, the method can more accurately determine the type of the reinforcement learning algorithm that matches the support capability of the device for the sampling process, and further improve the stability of the communication system in operation.

[0048] In an optional implementation, the communication function is power control, the communication operation is transmitting a communication signal with a first power, the action parameter is the first power, the state parameter is a path loss between the first device and the second device, and the reward parameter is a signal-to-interference-and-noise ratio of the communication signal.

[0049] In an optional implementation, the communication function is link adaptation, the communication operation is transmitting a communication signal with a first modulation and coding scheme, the action parameter is the first modulation and coding scheme, the state parameter includes channel state information (CSI) between the first device and the second device and a historical modulation and coding scheme, and the reward parameter is a transport block size of the communication signal.

[0050] In an optional implementation, the method further includes:

[0051] sending, to the second device, sampling capability information of the first device.

[0052] In an optional implementation, the method further includes:

[0053] receiving a first indication message, where the first indication message is used to indicate the use of the reinforcement learning algorithm.

[0054] In an optional implementation, the method further includes:

[0055] selecting the reinforcement learning algorithm according to the type of the reinforcement learning algorithm, and executing the reinforcement learning algorithm.

[0056] In a third aspect, an embodiment of the present application provides a method for selecting a reinforcement learning algorithm, applied to a second device, and the method includes:

[0057] determining sampling capability information of the second device, wherein the sampling capability information is information used to represent a support capability of the second device for a sampling process, the sampling process is a process in which the first device communicates with the second device to obtain the sampling content, the sampling capability information is used to determine a type of the reinforcement learning algorithm, the type of the reinforcement learning algorithm includes a hetero-policy type or a homo-policy type, the reinforcement learning algorithm defines the sampling process, and the reinforcement learning algorithm is used to implement the communication function;

[0058] sending, to the first device, the sampling capability information of the second device.

[0059] In view of the fact that different types of reinforcement learning algorithms correspond to different sampling characteristics (i.e., the homo-policy algorithm has a large number of sampling times and small storage requirements, and the hetero-policy algorithm has a small number of sampling times and large storage requirements), in the above method, the second device can send information representing the support capability of the second device for the sampling process to the first device, which is conducive to the subsequent determination by the first device of the type of the reinforcement learning algorithm that matches the support capability of the first device and / or the second device for the sampling process, and the execution of the reinforcement learning algorithm corresponding to the type, thereby reducing the impact of the sampling process on the stability of the communication system during operation, and further improving the stability of the communication system during operation.

[0060] In an optional implementation, the sampling content includes an action parameter, a state parameter, and a reward parameter, the action parameter is used to represent a feature of a communication operation performed by the first device on the second device in the sampling process, the state parameter is used to represent a loss of the communication operation, and the reward parameter is used to represent a network performance corresponding to the communication operation.

[0061] In an optional implementation, the sampling capability information includes one or more of a sampling frequency supported by the device, a sampling energy consumption, a storage capacity, an air interface load, an interference degree, and a transmission requirement.

[0062] In the above method, the support capability of the device for the sampling process represented by the sampling capability information includes a support capability of the device for the number of sampling times and a support capability of the device for the storage capacity required by the sampling, and because different types of reinforcement learning algorithms differ in the number of sampling times and the storage requirement, the above method can more accurately determine the type of the reinforcement learning algorithm that matches the support capability of the device for the sampling process, and further improve the stability of the communication system during operation.

[0063] In an optional implementation, the communication function is power control, the communication operation is the sending of a communication signal at a first power, the action parameter is the first power, the state parameter is a path loss between the first device and the second device, and the reward parameter is a signal-to-interference-and-noise ratio of the communication signal.

[0064] In an optional implementation, the communication function is link adaptation, the communication operation is transmitting a communication signal using a first modulation and coding scheme, the action parameter is the first modulation and coding scheme, the state parameter comprises channel state information (CSI) between the first device and the second device and a historical modulation and coding scheme, and the reward parameter is a transport block size of the transmitted communication signal.

[0065] In an optional implementation, the method further comprises:

[0066] receiving a first request message transmitted by the first device, the first request message being used to request the sampling capability information.

[0067] In an optional implementation, the method further comprises:

[0068] transmitting, to the first device, the sampling capability information of the third device.

[0069] In a fourth aspect, an embodiment of the present application provides a communication apparatus, which can be the first device in the first aspect or any possible implementation manner of the first aspect, wherein:

[0070] The communication apparatus comprises modules for performing the method in the first aspect or any possible implementation manner of the first aspect.

[0071] Or, the communication apparatus can be the first device in the second aspect or any possible implementation manner of the second aspect, wherein:

[0072] The communication apparatus comprises modules for performing the method in the second aspect or any possible implementation manner of the second aspect.

[0073] In a fifth aspect, an embodiment of the present application provides a communication apparatus, which can be the second device in the first aspect or any possible implementation manner of the first aspect, wherein:

[0074] The communication apparatus comprises modules for performing the method in the first aspect or any possible implementation manner of the first aspect.

[0075] Or, the communication apparatus can be the second device in the third aspect or any possible implementation manner of the third aspect, wherein:

[0076] The communication apparatus comprises modules for performing the method in the third aspect or any possible implementation manner of the third aspect.

[0077] In a sixth aspect, an embodiment of the present application provides a communication apparatus, which comprises a logic circuit and an interface, and the logic circuit and the interface are coupled; the interface is used to input and / or output information, wherein:

[0078] The logic circuit is configured to perform the method of the first aspect or any possible implementation of the first aspect.

[0079] Alternatively, the logic circuit is configured to perform the method of the second aspect or any possible implementation of the second aspect.

[0080] Alternatively, the logic circuit is configured to perform the method of the third aspect or any possible implementation of the third aspect.

[0081] In a seventh aspect, an embodiment of the present application provides a communication system, comprising a first device and a second device, wherein:

[0082] The first device is configured to perform the method performed by the first device in the first aspect or any possible implementation of the first aspect, and the second device is configured to perform the method performed by the second device in the third aspect or any possible implementation of the third aspect.

[0083] Alternatively, the first device is configured to perform the method performed by the first device in the second aspect or any possible implementation of the second aspect, and the second device is configured to perform the method performed by the second device in the first aspect or any possible implementation of the first aspect.

[0084] In an eighth aspect, an embodiment of the present application provides a computer readable storage medium configured to store a computer program, wherein:

[0085] The computer program, when executed, is configured to implement the method of the first aspect or any possible implementation of the first aspect.

[0086] Alternatively, the computer program, when executed, is configured to implement the method of the second aspect or any possible implementation of the second aspect.

[0087] Alternatively, the computer program, when executed, is configured to implement the method of the third aspect or any possible implementation of the third aspect.

[0088] The method, system and device provided by any possible implementation of the fourth aspect to the eighth aspect of the present application have the beneficial effects of the technical solutions provided by the first aspect, any possible implementation of the first aspect, the second aspect, any possible implementation of the second aspect, the third aspect and any possible implementation of the third aspect, which will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0089] The drawings used by the embodiments of the present application are described below.

[0090] Figure 1Ais a schematic diagram of an architecture of a communication system provided by an embodiment of the present application;

[0091] Figure 1A is a schematic diagram of a structure of an O-RAN device provided by an embodiment of the present application;

[0092] Figure 2 is a schematic diagram of a sampling process provided by an embodiment of the present application;

[0093] Figure 3 is a schematic diagram of an architecture of a communication system provided by an embodiment of the present application;

[0094] Figure 4 is a flowchart of a selection method of a reinforcement learning algorithm provided by an embodiment of the present application;

[0095] Figure 5 is a flowchart of a selection method of a reinforcement learning algorithm provided by an embodiment of the present application;

[0096] Figure 6 is a flowchart of a selection method of a reinforcement learning algorithm provided by an embodiment of the present application;

[0097] Figure 7 is a flowchart of a selection method of a reinforcement learning algorithm provided by an embodiment of the present application;

[0098] Figure 8 is a flowchart of a selection method of a reinforcement learning algorithm provided by an embodiment of the present application;

[0099] Figure 9 is a flowchart of a selection method of a reinforcement learning algorithm provided by an embodiment of the present application;

[0100] Figure 10 is a flowchart of a selection method of a reinforcement learning algorithm provided by an embodiment of the present application;

[0101] Figure 11 is a flowchart of a selection method of a reinforcement learning algorithm provided by an embodiment of the present application;

[0102] Figure 12 is a schematic diagram of a structure of a communication device provided by an embodiment of the present application;

[0103] Figure 13 is a schematic diagram of a structure of a communication device provided by an embodiment of the present application;

[0104] Figure 14 is a schematic diagram of a structure of a communication device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0105] The embodiments of this application are described below with reference to the accompanying drawings.

[0106] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. In the textual description of this application, the character " / " generally indicates an "or" relationship between the preceding and following related objects; in the formulas of this application, the character " / " indicates a "division" relationship between the preceding and following related objects. "Including at least one of A, B, and C" can mean: including A; including B; including C; including A and B; including A and C; including B and C; including A, B, and C.

[0107] The technical solutions provided in this application can be applied to various communication systems, such as Long Term Evolution (LTE) systems, LTE frequency division duplex (FDD) systems, LTE time division duplex (TDD) systems, 5th generation (5G) systems, or new radio (NR) systems. Furthermore, they can also be applied to subsequent evolution systems, such as 6G communication systems. These systems can be classified according to their operating modes into frequency division duplex (FDD) systems and time division duplex (TDD) systems.

[0108] The system architecture used in the embodiments of this application is described below. It should be noted that the system architecture and business scenarios described in this application are for the purpose of more clearly illustrating the technical solutions of this application, and do not constitute a limitation on the technical solutions provided in this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in this application are also applicable to similar technical problems.

[0109] For ease of understanding, Figure 1A This is a schematic diagram of the architecture of a communication system provided in an embodiment of this application. Figure 1A As shown, the communication system 10 includes a radio access network (RAN) 100 and a core network (CN) 200. The RAN 100 includes at least one RAN device (such as...). Figure 1A 110a and 110b, collectively referred to as 110) and at least one user equipment (such asFigure 1A RAN 100 (120a-120j, collectively referred to as 120) may also include other RAN equipment, such as wireless relay equipment, wireless backhaul equipment, and new equipment introduced in future networks (e.g., mobile intelligent devices responsible for wireless intelligence functions). Figure 1A (not shown in the image) etc. CN 200 includes at least one CN device 210.

[0110] User equipment 120 is connected to RAN equipment 110 wirelessly, for example, via air interface technology (such as NR or LTE). RAN equipment 110 is connected to CN 200 wirelessly or via a wired connection. CN equipment 210 in CN 200 and RAN equipment 110 in RAN 100 can be different physical devices, or they can be the same physical device integrating core network logical functions and radio access network logical functions.

[0111] RAN100 can be a cellular system related to the 3rd Generation Partnership Project (3GPP), such as 4G, 5G mobile communication systems, or future-oriented evolution systems. RAN100 can also be an open access network (O-RAN or ORAN), a cloud radio access network (CRAN), or a wireless fidelity (WiFi) system. RAN100 can also be a communication system that integrates two or more of the above systems.

[0112] RAN equipment 110 forms part of the communication system, assisting user equipment in achieving wireless access. RAN equipment can also be called RAN network elements, RAN entities, or RAN nodes, etc. Multiple RAN equipment 110 in communication system 10 can be of the same type or different types. In some scenarios, the roles of RAN equipment 110 and user equipment 120 are relative, for example, Figure 1A Network element 120i can be a helicopter or a drone, and it can be configured as a mobile base station. For user equipment 120j that accesses RAN 100 through network element 120i, network element 120i is a base station; however, for base station 110a, network element 120i is user equipment. RAN equipment 110 and user equipment 120 are sometimes referred to as communication equipment, for example... Figure 1A Network elements 110a and 110b can be understood as communication equipment with base station functions, while network elements 120a-120j can be understood as communication equipment with user equipment functions.

[0113] In an optional implementation, the RAN device can be a base station (BS), an evolved NodeB (eNodeB), an access point (AP), a transmission reception point (TRP), a next generation NodeB (gNB), a base station in a future mobile communication system, an access node in a WiFi system, or the like. The RAN device can be a macro base station (such as 110a in FIG. 1), a micro base station, or an indoor station (such as 110b in FIG. 1), a relay node or a donor node, or a wireless controller in a CRAN scenario. Optionally, the RAN device can also be a server, a wearable device, a vehicle or a vehicle-mounted device, or the like. For example, the access network device in vehicle to everything (V2X) technology can be a road side unit (RSU). All or part of the functions of the RAN device in this application can also be implemented by software functions running on hardware, or by virtualized functions instantiated on a platform (such as a cloud platform). The RAN device can also be provided with a communication module, circuit or chip for performing corresponding communication functions, and program instructions for performing corresponding communication functions. The RAN device in this application can also be a logical node, a logical module or software that can implement all or part of the functions of the RAN device. Figure 1A Figure 1A In an optional implementation, the RAN device can be a base station (BS), an evolved NodeB (eNodeB), an access point (AP), a transmission reception point (TRP), a next generation NodeB (gNB), a base station in a future mobile communication system, an access node in a WiFi system, or the like. The RAN device can be a macro base station (such as 110a in FIG. 1), a micro base station, or an indoor station (such as 110b in FIG. 1), a relay node or a donor node, or a wireless controller in a CRAN scenario. Optionally, the RAN device can also be a server, a wearable device, a vehicle or a vehicle-mounted device, or the like. For example, the access network device in vehicle to everything (V2X) technology can be a road side unit (RSU). All or part of the functions of the RAN device in this application can also be implemented by software functions running on hardware, or by virtualized functions instantiated on a platform (such as a cloud platform). The RAN device can also be provided with a communication module, circuit or chip for performing corresponding communication functions, and program instructions for performing corresponding communication functions. The RAN device in this application can also be a logical node, a logical module or software that can implement all or part of the functions of the RAN device.

[0114] In an optional implementation, the RAN device can be a base station (BS), an evolved NodeB (eNodeB), an access point (AP), a transmission reception point (TRP), a next generation NodeB (gNB), a base station in a future mobile communication system, an access node in a WiFi system, or the like. The RAN device can be a macro base station (such as 110a in FIG. 1), a micro base station, or an indoor station (such as 110b in FIG. 1), a relay node or a donor node, or a wireless controller in a CRAN scenario. Optionally, the RAN device can also be a server, a wearable device, a vehicle or a vehicle-mounted device, or the like. For example, the access network device in vehicle to everything (V2X) technology can be a road side unit (RSU). All or part of the functions of the RAN device in this application can also be implemented by software functions running on hardware, or by virtualized functions instantiated on a platform (such as a cloud platform). The RAN device can also be provided with a communication module, circuit or chip for performing corresponding communication functions, and program instructions for performing corresponding communication functions. The RAN device in this application can also be a logical node, a logical module or software that can implement all or part of the functions of the RAN device.

[0115] ​The CU (or CU-CP and CU-UP), DU or RU can also have different names in different systems, but those skilled in the art can understand their meanings. For example, in the ORAN system, the CU can also be referred to as O-CU (open CU), the DU can also be referred to as O-DU, the CU-CP can also be referred to as O-CU-CP, the CU-UP can also be referred to as O-CU-UP, and the RU can also be referred to as O-RU. For the convenience of description, the CU, CU-CP, CU-UP, DU and RU are taken as examples for description in this application. Any one of the CU (or CU-CP, CU-UP), DU and RU in this application can be implemented by a software module, a hardware module, or a combination of a software module and a hardware module.

[0116] Optionally, when the RAN 100 is an O-RAN, Figure 1A The RAN device 100 shown can be an O-RAN device 410 as shown in Figure 1A The RAN device 100 shown can be an O-RAN device 410 as shown in Figure 1BAs shown, the O-RAN device 410 includes a Non-Real-Time Radio access network Intelligent Controller (Non-RT RIC) device 4101, a Near-Real-Time Radio access network Intelligent Controller (Near-RT RIC) device 4102, an O-CU-CP 4103, an O-CU-UP 4104, an O-DU 4105, at least one O-RAN evolved NodeB (O-eNB) 4106. A user equipment (UE) 120 can access the above communication system and have a device or module with corresponding communication functions. The user equipment 120 can be widely applied in various scenarios, such as device-to-device (D2D), vehicle to everything (V2X) communication, machine-type communication (MTC), internet of things (IOT), virtual reality, augmented reality, industrial control, automatic driving, remote medical treatment, smart power grid, smart furniture, smart office, smart wear, smart transportation, smart city, etc. The user equipment can be a mobile phone, a tablet computer, a computer with wireless transceiver function, a wearable device, a vehicle, a drone, a helicopter, an airplane, a ship, a robot, a mechanical arm, a smart home device, a transport vehicle with wireless communication function, a communication module, etc. Embodiments of the present application do not limit the device form of the user equipment. The user equipment is usually provided with a communication module, circuit or chip for executing corresponding communication functions. The user equipment is also configured with program instructions for executing corresponding communication functions. The user equipment can also be referred to as a terminal, a terminal device, a mobile station, a mobile terminal, etc.

[0117] The CN device 210 in the CN 200 can implement one or more network functions, such as a mobility management entity (MME), an access and mobility management function (AMF), a session management function (SMF), a user plane function (UPF), a policy control function (PCF), a policy and charging rules function (PCRF), an edge application server discovery function (EASDF), a unified data management (UDM), a unified data repository (UDR), a home subscriber server (HSS), a centralized network configuration (CNC), a network repository function (NRF), a network exposure function (NEF), a local NEF (or L-NEF), a binding support function (BSF), an application function (AF), and the like. The CN device can also be referred to as a CN node, a CN network element, a CN entity, and the like.

[0118] With the application of reinforcement learning in communication systems, the communication system can implement communication functions such as access control, link adaptation, power control, interference coordination, load balancing, and mobility management.

[0119] Reinforcement learning is an important machine learning method. Reinforcement learning defines two types of roles, namely an agent and an environment. In reinforcement learning, the agent interacts with the environment, i.e., the agent observes the state of the environment, then selects an action according to the corresponding policy and executes it, the environment gives a reward according to the action of the agent, and enters a new state. Then, the agent adjusts the target policy according to the reward of the environment to obtain the highest cumulative reward.

[0120] Reinforcement learning can be modeled as a Markov Decision Process (MDP). A Markov Decision Process is a time-dependent ongoing process that can be composed of a state space S (also referred to as a state set), an action space A (also referred to as an action set), a state transition function P(s'|s,a), and a reward function r(s,a).

[0121] The interaction process between the agent and the environment in reinforcement learning can also be referred to as a sampling process. Based on the Markov Decision Process, the sampling process can be described as the process shown in Figure 1B Figure 2 Figure 2 is a schematic diagram of a sampling process provided by an embodiment of the present application. As shown in Figure 2 in the sampling process, the agent decides the action a t taken at time t according to the state s t of the environment at time t, the environment gives feedback r t (i.e., reward r t ) according to the action a t , and due to the action a t , the environment is converted from state s t to state s t+1 at time t+1. Wherein, the state s t and the state s t+1 belong to the state space S, and the action a t belongs to the action space A. It can be understood that the sampling process is the process of the agent interacting with the environment to obtain sampling content, which includes the action parameter a t , the state parameter s t and the reward parameter r t at time t.

[0122] The algorithm of reinforcement learning (hereinafter referred to as reinforcement learning algorithm) includes a sampling policy and a target policy. The sampling policy is the policy used in the sampling process. The target policy is the policy that the agent needs to learn, i.e., selecting the optimal action according to the current state. There are many reinforcement learning algorithms, which are usually divided into two categories, i.e., the same policy type reinforcement learning algorithm (hereinafter referred to as the same policy algorithm) and the different policy type reinforcement learning algorithm (hereinafter referred to as the different policy algorithm).

[0123] ​​The sampling policy in the on-policy algorithm is the same as the target policy. Optionally, the on-policy algorithm includes a state-action-reward-state-action (SARSA) algorithm, an advantage actor-critic (A2C) algorithm, and the like. For example, in the SARSA algorithm, both the sampling policy and the target policy are greedy policies. Moreover, the agent executing the on-policy algorithm must resample and only use the newly sampled data to update the value function each time the value function is updated during the learning process. Therefore, the agent executing the on-policy algorithm samples more frequently (sampling each time the value function is updated), but has less storage overhead (the agent does not need to save previously sampled data). It should be noted that one sampling process can include one or more rounds of interaction between the agent and the environment, which is not strictly limited in the embodiments of the present application.

[0124] The sampling policy in the off-policy algorithm is the same as the target policy. Optionally, the off-policy algorithm includes a Q-learning algorithm and a deep Q-network (DQN) algorithm, and the like. For example, in the Q-learning algorithm, the sampling policy is an ε-greedy policy, and the target policy is a greedy policy. Moreover, the agent executing the off-policy algorithm uses previously sampled data each time the value function is updated during the learning process, and does not need to sample and use new data, or only needs to sample and use a small amount of new data. Therefore, the agent executing the off-policy algorithm samples less frequently, but has more storage overhead (needs to save previously sampled data).

[0125] That is, the sampling characteristics of the on-policy algorithm and the off-policy algorithm can be shown in Table 1:

[0126] Table 1: Sampling characteristics of reinforcement learning algorithms

[0127] Figure 2 Algorithm type Sampling feature Same strategy Many sampling times, small storage requirement Different strategy

[0128] Further, when the reinforcement learning is applied in a communication system, the agent can be a communication device, and the environment can be understood as the communication system in which the communication device is located. That is, the environment at least includes the communication device and another device in the communication system that establishes a communication connection with the communication device. For the sake of distinction, hereinafter, the communication device as the agent is referred to as a first device, and the other device that establishes a communication connection with the communication device is referred to as a second device. Optionally, the environment can also include other communication devices in the communication system in addition to the first device and the second device, and for the sake of description, hereinafter, the other communication devices are referred to as third devices. The third devices can communicate directly or indirectly with the second devices.

[0129] For example, when reinforcement learning is applied in Few sampling times, large storage requirement In the communication system 10 shown, Figure 1A The communication system 10 shown can be simplified as follows: Figure 1A The communication system 10 is shown. Please refer to [link / reference]. Figure 3 , Figure 3 This is a schematic diagram of the architecture of another communication system provided in the embodiments of this application. Figure 3 The communication system 10 shown includes a first device 301 and a second device 302 that have established a communication connection. Optionally, the communication system 10 also includes a third device 303.

[0130] Optionally, the first device 301 can be Figure 3 User equipment 120a and second equipment 302 can be... Figure 1A Base station 110a, and third device 303 can be Figure 1A CN device 210 or Figure 1A Network management device not shown.

[0131] Optionally, the first device 301 can be Figure 1A The base station 110a and the second device 302 can be... Figure 1A The third device can be user equipment 120e, which is user equipment 120a.

[0132] Optionally, the first device 301 can be Figure 1A The base station 110a shown, the second device 302 can be Figure 1A In base station 110b, the third device can be... Figure 1A User equipment 120f in the middle.

[0133] Optionally, the first device 301 can be Figure 1A The second device 302 can be a radio access network intelligent controller (RIC) device (such as Non-RT RIC device 4101 or Near-RT RIC device 4102). Figure 1A In the O-eNB 4106, the third device 303 can be Figure 1B User equipment 120a in the middle.

[0134] The first device 301 is used to execute reinforcement learning algorithms. The interaction process between the agent and the environment (i.e., the sampling process) is the process by which the first device 301 communicates with the second device 302 to obtain sampled content. The action parameter 'a' in the sampled content... t The state parameter s is used to characterize the communication operation performed by the first device 301 against the second device 302 during the sampling process.t The reward parameter r is used to characterize the loss of this communication operation. t This is used to characterize the network performance corresponding to the communication operation. Optionally, the sampling process may also include a process in which the first device 301 communicates with the third device 303 through the second device 302 to obtain the sampled content.

[0135] Understandably, reinforcement learning algorithms can define the parameters (such as action parameters a) in the sampled content obtained during the sampling process. t State parameter s t and reward parameter r t The specific content corresponding to (etc.) is related to the communication function implemented by the reinforcement learning algorithm and can be set according to actual needs.

[0136] The following section provides an example of the defined sampling content in specific application scenarios.

[0137] Scenario 1: The reinforcement learning algorithm is used to implement uplink power control, and the first device 301 is... Figure 1B User equipment 120a and second equipment 302 are... Figure 1A Base station 110a.

[0138] During the sampling process, the communication operation performed by user equipment 120a on base station 110a is defined as sending a communication signal of a certain power. The action parameter 'a' in the sampling content... t Defined as a t =P t P t The transmission power at time t; state parameter s t Defined as st = PLT, PL t Let be the path loss (PL) between user equipment 120a and base station 110a at time t; and let r be the reward parameter. t Defined as r t =SINR t SINR t Let t be the signal-to-interference-plus-noise ratio (SINR) of the communication signal received by base station 110a at time t.

[0139] It is understandable that the target policy π(s) that user equipment 120a needs to learn is defined as π(s) t ) = optimala t This means selecting the optimal action based on the current state (which can also be understood as selecting the optimal transmit power for this transmission based on the current path loss). User equipment 120a can use the same strategy algorithm, in which case the sampling strategy π sample(s) is the same as the target policy π(s) described above. The user equipment 120a can also employ a heterogeneous policy algorithm, in which case the sampling policy π sample (s) is different from the target policy π(s) described above, for example, the sampling policy π sample (s) can be expressed as follows:

[0140]

[0141] That is, the sampling policy π sample (s) can define that the target policy π(s) is employed with a probability of 1-ε and other strategies (random selection strategies) are employed with a probability of ε.

[0142] Scenario two: the communication function implemented by the reinforcement learning algorithm is link adaptation, the first device 301 is the base station 110a in Figure 1A , and the second device 302 is the user equipment 120a in Figure 1A .

[0143] In the sampling process, the communication operation performed by the base station 110a for the user equipment 120a is defined as transmitting a communication signal employing a certain modulation and coding scheme (MCS). The action parameter a t in the sampling content is defined as a t =MCS t , where MCS t is the modulation and coding scheme at time t; the state parameter s t is defined as s t =(CSI t , MCS t-1 ), where CSI t is the channel state information (CSI) between the user equipment 120a and the base station 110a at time t, and MCS t-1 is the MCS employed by the base station 110a when transmitting a signal to the user equipment 120a at time t-1; the reward parameter r t is defined as r t =TBS t , where TBS t is the transport block size (TBS) of the signal transmitted by the base station 110a to the user equipment 120a at time t.

[0144] It can be understood that the target policy π(s) to be learned by the base station 110a is defined as π(s t ) = optimala tThat is, the optimal action is selected according to the current state (which can also be understood as selecting the optimal MCS for this transmission according to the current CSI and the MCS used in the last transmission). In the same scenario one, the base station 110a can use the same strategy algorithm or the different strategy algorithm, which will not be described here.

[0145] Further, in the communication system, the device for determining which type of reinforcement learning algorithm is used by the first device is usually a device for selecting the type of algorithm. The device for selecting the type of algorithm can be the first device or a component in the first device, or the second device or a component in the second device, or the third device or a component in the third device, and the embodiments of the present application do not make strict restrictions. In the above scenarios one and two, only the first device (i.e., the agent) is taken as an example for illustrative description of the device for selecting the type of algorithm.

[0146] At present, the device for selecting the type of algorithm usually determines which type of reinforcement learning algorithm is used by the first device according to the type of reinforcement learning algorithm supported by the first device. For example, when it is determined that the first device supports two types of reinforcement learning algorithms (e.g., it is determined that two types of reinforcement learning algorithms are stored in the first device), the device for selecting the type of algorithm randomly or by default selects one type from the two types, so that the first device executes the reinforcement learning algorithm of the type.

[0147] However, the sampling process in the reinforcement learning algorithm can affect the stability of the communication system during operation. For example, in the sampling process of scenario one, the user equipment 120a selects the transmission power P t Before the user equipment 120a learns the optimal target strategy according to the sampling content, the transmission power P t may not be the optimal value, so that the situation that the transmission power is too large, the energy consumption of the user equipment 120a is too large, or the reception of other communication signals by the base station 110a is interfered may occur. As can be seen from Table 1, if the reinforcement learning algorithm executed by the user equipment 120a is the same strategy algorithm, the number of samplings is large, and the influence on the stability of the communication system during operation is large.

[0148] For example, in the sampling process of scenario two, the base station 110a needs to store the obtained sampling content. As can be seen from Table 1, if the reinforcement learning algorithm executed by the base station 110a is the different strategy algorithm, the base station 110a needs to store more historical sampling content, so that the situation that the base station 110a cannot normally execute the reinforcement learning algorithm due to insufficient storage capacity may occur, thereby affecting the stability of the communication system during operation.

[0149] In view of this, the selection method and related device of the reinforcement learning algorithm according to the embodiments of the present application can improve the stability of the communication system during operation.

[0150] Please see Figure 1A , Figure 1A This is a flowchart illustrating a method for selecting a reinforcement learning algorithm provided in an embodiment of this application. This method can be based on... Figure 4 or Figure 4 The architecture shown can be used to implement this method, but it can also be implemented based on other architectures. This method includes, but is not limited to, the following steps:

[0151] Step S401: Select the device for algorithm type selection and obtain the sampling capability information of the first device and / or the second device.

[0152] As mentioned earlier, the algorithm type selection device is a device capable of determining the type (i.e., same-policy type or different-policy type) of the reinforcement learning algorithm to be executed by the first device. The first device is a device capable of storing and executing reinforcement learning algorithms, and the second device is a device capable of directly communicating with the first device. During the sampling process of the reinforcement learning algorithm definition, the first device and the second device communicate to obtain the sampling content. For detailed explanations of the algorithm type selection device, the first device, and the second device, please refer to the preceding text. Figure 1A The descriptions of the corresponding parts in the illustrated embodiments, and the specific explanations of the sampling process and sampling content, can be found in the previous explanation of the reinforcement learning algorithm, and will not be repeated here.

[0153] Sampling capability information characterizes a device's ability to support the sampling process. This capability can include the device's ability to support a certain number of samples and its ability to support the storage capacity required for sampling. In essence, sampling capability information reflects the resources available to the device for the sampling process, the overhead the device can withstand during the sampling process, or the device's capacity to withstand the impact of the sampling process.

[0154] In one alternative implementation, the sampling capability information may include one or more of the following: the sampling frequency supported by the device, sampling power consumption, storage capacity, air interface load, interference level, and transmission requirements.

[0155] For example, the sampling frequency supported by the device can be the maximum sampling frequency supported by the device (e.g., 5 samples per minute), the maximum sampling frequency level supported by the device (e.g., the level can be high, medium, or low, and the sampling frequency range corresponding to each level can be specified in the standard), or the minimum time interval between two adjacent sampling processes supported by the device (e.g., the minimum time interval is 10 seconds). It can be understood that the higher the frequency, the higher the frequency level, or the smaller the minimum time interval mentioned above, the stronger the device's ability to support the number of samples.

[0156] For example, the energy consumption supported by the device can be the energy consumed by the device in each sampling process (e.g., 100 kilojoules), the level of energy consumed by the device in each sampling (e.g., the level can be high, medium, or low), the additional energy consumed by the device in each sampling process, or the level of additional energy consumed by the device in each sampling process, etc. It should be noted that the additional energy consumed by the device is the energy consumed by the device only for the operation performed in the sampling process. For example, in the above scenario one, the base station 110a transmits the reward parameter r t to the user equipment 120a, and the energy consumed by the base station 110a is the additional energy consumed by the base station 110a (the base station 110a does not transmit the reward parameter r t in the normal communication process). It can be understood that the greater the above-mentioned energy or the higher the level of energy, the higher the overhead (energy consumption) that the device can bear, that is, the stronger the support capability of the device for the number of samplings.

[0157] For example, the storage capacity supported by the device can be the total storage capacity of the device (e.g., 1 GB, GB is gigabyte), the remaining storage capacity of the device (e.g., 500 MB, MB is megabyte), the capacity available for storing sampling content (also referred to as sampling samples) of the device (e.g., 300 MB), the proportion of the remaining capacity of the device to the total capacity (e.g., 50%), the proportion of the capacity available for storing sampling content of the device to the total capacity (e.g., 30%), the number of samples that can be stored by the device (e.g., 1000), etc.

[0158] It can be understood that a sample includes the sampling content collected in one sampling process. The higher the above-mentioned storage capacity, the higher the proportion, or the greater the number of samples, the stronger the support capability of the device for the storage capacity required for sampling. Alternatively, the strength of the support capability of the device for the storage capacity required for sampling can be measured by a preset storage capacity threshold. For example, if the energy consumed by the device in each sampling process is greater than the preset energy consumption threshold, the algorithm type selection device can determine that the support capability of the device for the number of samplings is strong.

[0159] For example, the air interface load supported by the device can be the utilization rate of the air interface physical resource block (PRB) of the device (e.g., 50%) or the air interface load level (e.g., high, medium, or low), etc. It can be understood that in the sampling process, the information interaction between the first device and the second device occupies the air interface transmission resource, and the lower the utilization rate of the air interface physical resource block of the device or the lower the air interface load level, the stronger the support capability of the device for the number of samplings.

[0160] For example, the interference level supported by the device can be the maximum (or average) of the interference suffered by the plurality of other devices that establish a connection with the device, or the maximum (or average) level of the interference suffered by the plurality of other devices that establish a connection with the device. For example, in the above-described scenario one, the other user devices and the neighboring base station that establish a connection with the base station 110a can suffer signal interference when the user device 120a transmits a communication signal to the base station 110a. It can be understood that the greater the maximum or the higher the maximum level of the interference suffered by the plurality of other devices that establish a connection with the device, the higher the interference bearing capacity of the device, i.e., the stronger the support capability of the device for the sampling times.

[0161] For example, the transmission requirement supported by the device can be a transmission requirement satisfaction ratio of the device (for example, the ratio of the actual transmission rate to the required transmission rate is 80%, or the ratio of the required transmission delay to the actual transmission delay is 90%), or a transmission requirement satisfaction level of the device (for example, the level can be high, medium and low, and the specific meaning of each level can be specified in the standard). It can be understood that the higher the ratio of the actual transmission rate to the required transmission rate of the device, the lower the ratio of the required transmission delay to the actual transmission delay of the device, and the higher the transmission requirement satisfaction of the device, the higher the transmission capacity of the device, i.e., the stronger the support capability of the device for the sampling times.

[0162] Optionally, the strength of the support capability of the device for the sampling times and the strength of the support capability of the device for the storage capacity required by the sampling can be measured by a preset judgment condition for the sampling capability information. For example, the preset judgment condition can define that when the maximum sampling frequency supported by the device is greater than a preset frequency threshold, the algorithm type selection device can determine that the support capability of the device for the sampling times is strong. For another example, the preset judgment condition can also define that when the energy consumed by the device in each sampling process is greater than a preset energy threshold, the algorithm type selection device can determine that the support capability of the device for the sampling times is strong. For another example, the preset judgment condition can also define that when the total storage capacity of the device is greater than a preset storage capacity, the algorithm type selection device can determine that the support capability of the device for the storage capacity required by the sampling is strong.

[0163] In addition, in actual applications, the communication function implemented by the reinforcement learning algorithm can be combined to determine which or which several pieces of information in the above examples the sampling capability information contains, and embodiments of the present application do not strictly limit this. For example, if the communication function implemented by the reinforcement learning algorithm is uplink power control, more attention can be paid to the anti-interference capability of the device; if the communication function implemented by the reinforcement learning algorithm is link adaptation, more attention can be paid to the transmission requirements supported by the device. In addition, in actual applications, the sampling capability information of the first device can be the same as or different from the sampling capability information of the second device, and embodiments of the present application do not strictly limit this. For example, since the first device is an agent, more attention can be paid to the sampling frequency and / or storage capacity supported by the first device, and more attention can be paid to the air interface load and / or interference level supported by the second device.

[0164] In an optional embodiment, the algorithm type selection device can not only obtain the sampling capability information of the first device and the second device, but also obtain the sampling capability information of a third device. The third device is in the same communication system as the first device and the second device, and the third device can directly or indirectly communicate with the second device. For specific descriptions of the third device, please refer to the descriptions of the corresponding parts in the embodiments described above, which will not be repeated here. It can be seen that the algorithm type selection device can obtain the sampling capability information of all devices related to the sampling process in the communication system, i.e., obtain the sampling capability information of the communication system. Figure 3

[0165] Step S402: The algorithm type selection device determines the type of the reinforcement learning algorithm according to the sampling capability information.

[0166] As known from the foregoing, the type of the reinforcement learning algorithm includes the same policy type and the different policy type, and the sampling characteristics of different types of reinforcement learning algorithms are different (as shown in Table 1).

[0167] After the algorithm type selection device obtains the sampling capability information of the first device and / or the second device, the sampling characteristics that match the support capability of the device for the sampling process are selected, so as to determine the type of the reinforcement learning algorithm corresponding to the sampling characteristics.

[0168] It can be understood that if the algorithm type selection device determines according to the sampling capability information that the support capability of the device for the sampling times is strong and / or the support capability of the device for the storage capacity required by the sampling is weak, it can be further determined that the type of the reinforcement learning algorithm is the same policy type. If the algorithm type selection device determines according to the sampling capability information that the support capability of the device for the sampling times is weak and / or the support capability of the device for the storage capacity required by the sampling is strong, it can be further determined that the type of the reinforcement learning algorithm is the different policy type. For explanations and descriptions of the sampling capability information, please refer to the descriptions of the corresponding parts in step S401, which will not be repeated here. ​

[0169] In summary, considering different sampling features corresponding to different types of reinforcement learning algorithms, the embodiment of the present application can obtain the sampling capability information of at least one of the first device and / or the second device before the sampling process, and select the type of reinforcement learning algorithm matched with the support capability of the device to the sampling process represented by the sampling capability information, which is conducive to the first device to execute the type of reinforcement learning algorithm corresponding thereto, thereby reducing the impact of the sampling process on the stability of the communication system during operation, and further improving the stability of the communication system during operation.

[0170] Further, as described above, the algorithm type selection device in the embodiment of the present application can be the first device or a component in the first device, or the second device or a component in the second device, or the third device or a component in the third device. Hereinafter, two cases of the algorithm type selection device being the first device and the second device are taken as examples to further explain and describe the method flowchart shown in Figure 3 .

[0171] Case one: the algorithm type selection device is the first device.

[0172] For easy understanding, please refer to Figure 3 , Figure 2 is a flowchart of another method for selecting a reinforcement learning algorithm provided by the embodiment of the present application, which can be implemented based on the architecture shown in Figure 5 or Figure 5 , or implemented based on other architectures. The method includes but is not limited to the following steps:

[0173] Step S501: the first device sends a first request message to the second device.

[0174] The first request message is used to request the sampling capability information of the second device.

[0175] In an optional implementation, the first request message includes information reflecting the specific content of the definition of each parameter (such as the action parameter a t , the state parameter s t and the reward parameter r t , etc.) in the sampling content. As described above, the specific content is related to the communication function implemented by the reinforcement learning algorithm to be executed by the first device, which can be set according to actual needs. After receiving the first request message, the second device can determine the sampling information to be fed back.

[0176] Optionally, the first request message can include the specific content. For example, the reinforcement algorithm learning is used to implement the uplink power control, and the first request information includes the content "action parameter a t =P t , state parameter st = PL t , reward parameter r t = SINR t

[0177] Optionally, the first request message can comprise information indicating the type of the communication function to be implemented. Illustratively, the first device and the second device store a correspondence between the type of the communication function and the sampling content, and the first request message sent by the first device to the second device comprises the type of the communication function.

[0178] In an optional embodiment, the first device sends the first request message to the second device after receiving the first indication message. The first indication message is used to indicate that the reinforcement learning algorithm is used. Illustratively, the first indication message can be sent by another device in the communication system in which the first device is located, which can manage the first device.

[0179] In an optional embodiment, the first device is built-in with a strategy for triggering the reinforcement learning algorithm, which can define that the first device will send the first request message to the second device when a certain network performance index does not meet a corresponding preset threshold. Illustratively, the strategy can define that the first device will send the first request message to the second device when the throughput of the first device does not reach a preset throughput threshold.

[0180] Step S502: The second device sends the sampling capability information of the second device to the first device.

[0181] The second device sends the sampling capability information of the second device to the first device after receiving the first request message. For specific description of the sampling capability information, please refer to the description of the corresponding part in step S401 of the embodiment shown in Figure 1A

[0182] Step S503: The first device determines the type of the reinforcement learning algorithm according to the sampling capability information of the first device and the second device.

[0183] The first device can obtain the sampling capability information of the first device, and determine the type of the reinforcement learning algorithm according to the sampling capability information of the first device and the sampling capability information of the second device sent by the second device. For specific description of step S503, please refer to the description of the corresponding part in step S402 of the embodiment shown in Figure 3

[0184] In an optional embodiment, the first device can also obtain the sampling capability information of the third device through the second device. For specific description of the third device, please refer to the foregoing description. Figure 4 ​​​The descriptions of the corresponding parts in the embodiments shown are not repeated here. The first device can determine the type of the reinforcement learning algorithm according to the sampling capability information of the first device, the second device and the third device.

[0185] Step S504: The first device selects the reinforcement learning algorithm according to the type of the reinforcement learning algorithm, and executes the reinforcement learning algorithm.

[0186] After the first device determines the type of the reinforcement learning algorithm to be executed, the first device can select the reinforcement learning algorithm corresponding to the type and execute the reinforcement learning algorithm. In the embodiments of the present application, "executing the reinforcement learning algorithm" refers to sampling according to the sampling strategy of the reinforcement learning algorithm to optimize the target policy.

[0187] In an optional implementation, the first device stores multiple reinforcement learning algorithms belonging to the same type, and after the first device determines the type of the reinforcement learning algorithm to be executed, the first device selects one reinforcement learning algorithm from the multiple reinforcement learning algorithms according to factors such as algorithm complexity and / or algorithm performance to execute. For example, if the first device determines that the type of the reinforcement learning algorithm to be executed is the same policy type, and the first device stores a SARSA algorithm and an A2C algorithm belonging to the same policy type, the first device can select the SARSA algorithm with smaller algorithm complexity.

[0188] The following exemplary descriptions are made in combination with specific application scenarios. Figure 4 The embodiments shown are exemplary.

[0189] Scenario three: the first device is a user equipment (i.e., an algorithm type selection device), and the second device is a base station.

[0190] For ease of understanding, please refer to Figure 3 , Figure 5 is a flowchart of another method for selecting a reinforcement learning algorithm provided by the embodiments of the present application. The method can be implemented based on the architecture shown in Figure 6 or Figure 6 , or can be implemented based on other architectures. The method includes but is not limited to the following steps:

[0191] Step S601: The user equipment determines to use a reinforcement learning algorithm.

[0192] In an optional implementation, the user equipment determines to use a reinforcement learning algorithm after receiving a first indication message. The first indication message is used to indicate the use of the reinforcement learning algorithm.

[0193] In an optional embodiment, the user equipment has a policy for triggering the reinforcement learning algorithm. The policy can define that the user equipment determines to use the reinforcement learning algorithm when a certain network performance indicator does not meet a corresponding preset threshold. For example, the policy can define that the user equipment determines to use the reinforcement learning algorithm for implementing the uplink power control when the uplink signal power exceeds a preset power threshold.

[0194] Step S602: The user equipment sends a first request message to the base station.

[0195] After the user equipment determines to use the reinforcement learning algorithm, the user equipment sends the first request message to the base station.

[0196] For specific description of steps S601 and S602, refer to the description of the corresponding parts in step S501 of the embodiment shown in Figure 1A The description of the corresponding parts in step S501 of the embodiment shown in

[0197] Step S603: The base station sends the sampling capability information of the base station to the user equipment.

[0198] In an optional embodiment, the reinforcement learning algorithm is used for implementing the uplink power control, and the sampling capability information of the base station can include one or more of the sampling frequency, the sampling energy consumption, the air interface load, and the interference level supported by the base station.

[0199] For specific description of step S603, refer to the description of the corresponding parts in step S502 of the embodiment shown in Figure 3 The description of the corresponding parts in step S502 of the embodiment shown in

[0200] Step S604: The user equipment determines the type of the reinforcement learning algorithm according to the sampling capability information of the user equipment and the base station.

[0201] In an optional embodiment, the user equipment can also obtain the sampling capability information of a third device through the base station, and determine the type of the reinforcement learning algorithm according to the sampling capability information of the user equipment, the base station, and the third device. The third device can be a base station adjacent to the base station, a user equipment in communication with the base station adjacent to the base station, or another user equipment in communication with the base station, and the present embodiment does not make strict limitation thereto.

[0202] For specific description of step S604, refer to the description of the corresponding parts in step S503 of the embodiment shown in Figure 5 The description of the corresponding parts in step S503 of the embodiment shown in

[0203] Step S605: The user equipment selects the reinforcement learning algorithm according to the type of the reinforcement learning algorithm.

[0204] Step S606: The user equipment executes the reinforcement learning algorithm.

[0205] For specific description of step S605 or step S606, please refer to the description of the corresponding part in step S501 of the embodiment shown in Figure 5 For specific description of step S504 of the embodiment shown in

[0206] Scenario four: the first device is a base station (i.e., the algorithm type selection device), and the second device is a user equipment.

[0207] For better understanding, please refer to Figure 5 , Figure 6 is a flow diagram of another method for selecting a reinforcement learning algorithm provided by an embodiment of the present application. The method can be implemented based on the architecture shown in Figure 7 or Figure 7 , or based on other architectures. The method includes but is not limited to the following steps:

[0208] Step S701: The base station determines to use a reinforcement learning algorithm.

[0209] In an optional implementation, the base station determines to use a reinforcement learning algorithm after receiving a first indication message. The first indication message is used to indicate the use of a reinforcement learning algorithm.

[0210] In an optional implementation, the base station is built-in with a policy for triggering a reinforcement learning algorithm. The policy can define that the user equipment determines to use a reinforcement learning algorithm in the case that a certain network performance indicator does not meet a corresponding preset threshold. For example, the policy can define that the base station determines to use a reinforcement learning algorithm for implementing link adaptation in the case that the throughput of the base station does not reach a preset throughput threshold.

[0211] Step S702: The base station sends a first request message to the user equipment.

[0212] After the base station determines to use a reinforcement learning algorithm, the base station sends a first request message to the user equipment.

[0213] For specific description of step S701 and step S702, please refer to the description of the corresponding part in step S501 of the embodiment shown in Figure 1A For specific description of step S504 of the embodiment shown in

[0214] Step S703: The user equipment sends the sampling capability information of the user equipment to the base station.

[0215] In an optional implementation, the reinforcement learning algorithm is used to implement link adaptation, and the sampling capability information of the user equipment can include one or more of the sampling frequency supported by the user equipment, the sampling energy consumption, and the transmission requirement.

[0216] For specific description of step S703, please refer to the description of the corresponding part in step S501 of the embodiment shown in Figure 3The description of the corresponding part in step S502 of the illustrated embodiment will not be repeated here.

[0217] Step S704: The base station determines the type of the reinforcement learning algorithm according to the sampling capability information of the base station and the user equipment.

[0218] In an optional implementation, the base station can also obtain the sampling capability information of a third device through the user equipment, and determine the type of the reinforcement learning algorithm according to the sampling capability information of the user equipment, the base station and the third device. The third device can be another base station in communication with the user equipment, a user equipment in communication with the other base station, etc., and the embodiments of the present application do not make strict restrictions thereon.

[0219] For specific description of step S704, please refer to Figure 5 The description of the corresponding part in step S503 of the illustrated embodiment will not be repeated here.

[0220] Step S705: The base station selects the reinforcement learning algorithm according to the type of the reinforcement learning algorithm.

[0221] Step S706: The base station executes the reinforcement learning algorithm.

[0222] For specific description of step S705 or step S706, please refer to Figure 5 The description of the corresponding part in step S504 of the illustrated embodiment will not be repeated here.

[0223] Scenario five: the first device is a RIC device (i.e., an algorithm type selection device), the second device is a base station, and the third device is a user equipment.

[0224] For ease of understanding, please refer to Figure 5 , Figure 5 is a flow diagram of another method for selecting a reinforcement learning algorithm provided by the embodiments of the present application. The method can be implemented based on the architecture shown in Figure 8 or Figure 8 , or can be implemented based on other architectures. The method includes but is not limited to the following steps:

[0225] Step S801: The RIC device determines to use the reinforcement learning algorithm.

[0226] In an optional implementation, the RIC device determines to use the reinforcement learning algorithm after receiving a first indication message. The first indication message is used to indicate the use of the reinforcement learning algorithm.

[0227] In an optional implementation, the RIC device is built-in with a policy for triggering the reinforcement learning algorithm. The policy can define that the user equipment will determine to use the reinforcement learning algorithm in the case that a certain network performance indicator does not meet a corresponding preset threshold.

[0228] Step S802: The RIC device sends a first request message to the base station.

[0229] After the RIC device determines to use the reinforcement learning algorithm, the RIC device sends a first request message to the base station. The first request message is used to request the sampling capability information of the base station, and can also be used to request the sampling capability information of the user equipment in communication with the base station.

[0230] Step S803: The base station sends a second request message to the user equipment.

[0231] After receiving the first request message sent by the RIC device, the base station sends a second request message to the user equipment. The second request message is used to request the sampling capability information of the user equipment. It can be understood that, optionally, the second request message includes information reflecting the specific content defined for each parameter (such as the action parameter a t , the state parameter s t and the reward parameter r t , etc.), same as the first request message.

[0232] The specific description of steps S801 and S803 can refer to the description of the corresponding part in step S501 of the embodiment shown in Figure 1A , and will not be repeated here.

[0233] Step S804: The user equipment sends the sampling capability information of the user equipment to the base station.

[0234] In an optional implementation, the reinforcement learning algorithm is used to implement link adaptation, and the sampling capability information of the user equipment can include one or more of the sampling frequency, the sampling energy consumption and the transmission requirement supported by the user equipment, and the sampling capability information of the base station can include one or more of the sampling frequency, the sampling energy consumption, the air interface load and the interference level supported by the base station.

[0235] Step S805: The base station sends the sampling capability information of the base station and the user equipment to the RIC device.

[0236] The base station can obtain its own sampling capability information, and after receiving the sampling capability information of the user equipment, the base station sends the sampling capability information of the base station and the user equipment to the RIC device.

[0237] The specific description of steps S804 and S805 can refer to the description of the corresponding part in step S502 of the embodiment shown in Figure 3 , and will not be repeated here.

[0238] Step S806: The RIC device determines the type of the reinforcement learning algorithm according to the sampling capability information of the RIC device, the base station and the user equipment.

[0239] For specific description of step S806, refer to the description of the corresponding part in step S506 of the embodiment shown in Figure 5 For specific description of step S503 of the embodiment shown in

[0240] Step S807: The RIC device selects a reinforcement learning algorithm according to the type of the reinforcement learning algorithm.

[0241] Step S808: The RIC device executes the reinforcement learning algorithm.

[0242] For specific description of steps S807 to S808, refer to the description of the corresponding part in step S504 of the embodiment shown in Figure 5 For specific description of step S503 of the embodiment shown in

[0243] It should be noted that the operations, technical terms, and technical logics involved in steps S501 to S504, steps S601 to S606, steps S701 to S706, and steps S801 to S808 can refer to the related descriptions of the embodiments shown in Figure 5 For specific description of step S503 of the embodiment shown in Figure 5 For specific description of step S503 of the embodiment shown in Figure 4 For specific description of step S503 of the embodiment shown in

[0244] Case two: The algorithm type selection device is the second device.

[0245] For better understanding, refer to Figure 5 to Figure 8 , Figure 4 is a flow diagram of another method for selecting a reinforcement learning algorithm provided by the embodiments of the present application. The method can be implemented based on the architecture shown in Figure 9 or Figure 9 , or can be implemented based on other architectures. The method includes but is not limited to the following steps:

[0246] Step S901: The first device sends the sampling capability information of the first device to the second device.

[0247] In an optional implementation, after receiving the first indication message, the first device sends the sampling capability information of the first device to the second device. The first indication message is used to indicate the use of a reinforcement learning algorithm. For example, the first indication message can be sent by another device in the communication system in which the first device is located, which can manage the first device.

[0248] In an optional implementation, the first device is built with a policy for triggering the reinforcement learning algorithm, which can define that the first device will send the sampling capability information of the first device to the second device when a certain network performance index does not meet a corresponding preset threshold. For example, the policy can define that the first device will send the sampling capability information of the first device to the second device when the throughput of the first device does not reach a preset throughput threshold.

[0249] For specific description of the sampling capability information, refer to the description of the corresponding part in step S401 of the embodiment shown in Figure 1A For specific description of the sampling capability information, refer to the description of the corresponding part in step S401 of the embodiment shown in

[0250] Step S902: The second device determines the type of the reinforcement learning algorithm according to the sampling capability information of the first device and the second device.

[0251] After receiving the sampling content of the first device, the second device obtains the sampling content of the second device, and determines the type of the reinforcement learning algorithm according to the sampling capability information of the first device and the second device. For specific description of step S902, refer to the description of the corresponding part in step S402 of the embodiment shown in Figure 3 For specific description of the sampling capability information, refer to the description of the corresponding part in step S401 of the embodiment shown in

[0252] In an optional implementation, after receiving the sampling content of the first device, the second device can also send a second request message to the third device, where the second request message is used to request the sampling capability information of the third device. For explanation and description of the second request message, refer to the description of the corresponding part in step S803 of the embodiment shown in Figure 4 For specific description of the sampling capability information, refer to the description of the corresponding part in step S401 of the embodiment shown in

[0253] Step S903: The second device sends type information to the first device.

[0254] After determining the type of the reinforcement learning algorithm that the first device needs to execute, the second device sends type information to the first device. The type message includes the type of the reinforcement learning algorithm.

[0255] Step S904: The first device selects the reinforcement learning algorithm according to the type information, and executes the reinforcement learning algorithm.

[0256] For specific description of step S904, refer to the description of the corresponding part in step S504 of the embodiment shown in Figure 4 For specific description of the sampling capability information, refer to the description of the corresponding part in step S401 of the embodiment shown in

[0257] The embodiments shown in Figure 8 are exemplarily described in combination with specific application scenarios.

[0258] Scenario six: the first device is a user equipment, and the second device is a base station (i.e., the algorithm type selection device).

[0259] For ease of understanding, please refer to Figure 5 , Figure 9 is a flowchart of another method for selecting a reinforcement learning algorithm provided in the embodiments of the present application. The method can be implemented based on the architecture shown in Figure 10 or Figure 10 , or can be implemented based on other architectures. The method includes, but is not limited to, the following steps:

[0260] Step S1001: The user equipment determines to use a reinforcement learning algorithm.

[0261] In an optional implementation, the user equipment determines to use the reinforcement learning algorithm after receiving a first indication message. The first indication message is used to indicate the use of the reinforcement learning algorithm.

[0262] In an optional implementation, the user equipment is built-in with a policy for triggering the reinforcement learning algorithm. The policy can define that the user equipment determines to use the reinforcement learning algorithm in a case where a certain network performance indicator does not meet a corresponding preset threshold. For example, the policy can define that the user equipment determines to use the reinforcement learning algorithm for implementing uplink power control in a case where the uplink signal power exceeds a preset power threshold.

[0263] Step S1002: The user equipment sends sampling capability information of the user equipment to the base station.

[0264] After the user equipment determines to use the reinforcement learning algorithm, the user equipment sends the sampling capability information of the user equipment to the base station. In an optional implementation, the reinforcement learning algorithm is used to implement link adaptation, and the sampling capability information of the user equipment can include one or more of the sampling frequency, the sampling energy consumption, and the storage capacity supported by the user equipment.

[0265] For specific descriptions of steps S1001 and S1002, please refer to the descriptions of the corresponding parts in step S901 of the embodiment shown in Figure 1A .

[0266] Step S1003: The base station determines the type of the reinforcement learning algorithm based on the sampling capability information of the user equipment and the base station.

[0267] In an optional implementation, the base station can also obtain sampling capability information of a third device, and determine the type of the reinforcement learning algorithm based on the sampling capability information of the user equipment, the base station, and the third device. The third device can be a base station adjacent to the base station, a user equipment in communication with the base station adjacent to the base station, or another user equipment in communication with the base station, etc., which are not strictly limited by the embodiments of the present application.

[0268] For a detailed explanation of step S1003, please refer to Figure 3 The description of the corresponding part in step S902 of the illustrated embodiment will not be repeated here.

[0269] Step S1004: The base station sends type information to the user equipment.

[0270] After determining the type of reinforcement learning algorithm that the user equipment needs to execute, the base station sends type information to the user equipment. The type message includes the type of reinforcement learning algorithm.

[0271] Step S1005: The user equipment selects a reinforcement learning algorithm based on the type information.

[0272] Step S1006: The user equipment executes the reinforcement learning algorithm.

[0273] For detailed explanations of steps S1005 to S1006, please refer to [link / reference]. Figure 9 The description of the corresponding part in step S904 of the illustrated embodiment will not be repeated here.

[0274] Scenario 7: The first device is a base station, and the second device is a user equipment (i.e., the algorithm type selection device).

[0275] For better understanding, please refer to Figure 9 , Figure 9 This is a flowchart illustrating another reinforcement learning algorithm selection method provided in this application embodiment. This method can be based on... Figure 11 or Figure 11 The architecture shown can be used to implement this method, but it can also be implemented based on other architectures. This method includes, but is not limited to, the following steps:

[0276] Step S1101: The base station determines which reinforcement learning algorithm to use.

[0277] In one alternative implementation, after receiving the first indication message, the base station determines to use a reinforcement learning algorithm. The first indication message is used to indicate the use of a reinforcement learning algorithm.

[0278] In one alternative implementation, the base station has a built-in policy for triggering reinforcement learning algorithms. This policy can be defined so that the user equipment will determine to use a reinforcement learning algorithm when a certain network performance metric fails to meet a corresponding preset threshold. For example, the policy could define that if the base station's throughput does not reach a preset throughput threshold, the base station will determine to use a reinforcement learning algorithm for link adaptation.

[0279] Step S1102: The base station sends the base station's sampling capability information to the user equipment.

[0280] After the base station determines to use the reinforcement learning algorithm, the base station sends the sampling capability information of the base station to the user equipment. In an optional embodiment, the reinforcement learning algorithm is used to implement link adaptation, and the sampling capability information of the base station can include one or more of the sampling frequency, the sampling energy consumption, and the storage capacity supported by the base station.

[0281] For specific description of steps S1101 and S1102, refer to the description of the corresponding part in step S901 of the embodiment shown in Figure 1A For specific description of steps S1101 and S1102, refer to the description of the corresponding part in step S901 of the embodiment shown in

[0282] Step S1103: The user equipment determines the type of the reinforcement learning algorithm according to the sampling capability information of the user equipment and the base station.

[0283] In an optional embodiment, the user equipment can also obtain the sampling capability information of a third device, and determine the type of the reinforcement learning algorithm according to the sampling capability information of the user equipment, the base station, and the third device. The third device can be another base station in communication with the user equipment, a user equipment in communication with the another base station, etc., which are not strictly limited by the embodiments of the present application.

[0284] For specific description of step S1103, refer to the description of the corresponding part in step S902 of the embodiment shown in Figure 3 For specific description of step S1103, refer to the description of the corresponding part in step S902 of the embodiment shown in

[0285] Step S1104: The user equipment sends the type information to the base station.

[0286] After the user equipment determines the type of the reinforcement learning algorithm to be executed by the base station, the user equipment sends the type information to the base station. The type information includes the type of the reinforcement learning algorithm.

[0287] Step S1105: The base station selects the reinforcement learning algorithm according to the type information.

[0288] Step S1106: The base station executes the reinforcement learning algorithm.

[0289] For specific description of steps S1105 to S1106, refer to the description of the corresponding part in step S904 of the embodiment shown in Figure 9 For specific description of steps S1105 to S1106, refer to the description of the corresponding part in step S904 of the embodiment shown in

[0290] It should be noted that the operations, technical terms, and technical logics involved in steps S901 to S904, steps S1001 to S1006, and steps S1101 to S1106 can refer to the related description of the embodiments shown in Figure 9 For specific description of steps S1105 to S1106, refer to the description of the corresponding part in step S904 of the embodiment shown in Figure 9 For specific description of steps S1105 to S1106, refer to the description of the corresponding part in step S904 of the embodiment shown in Figure 4 For specific description of steps S1105 to S1106, refer to the description of the corresponding part in step S904 of the embodiment shown in

[0291] The communication apparatus provided by the embodiments of the present application will be described below.

[0292] Figure 9 to Figure 11 is a structural schematic diagram of a communication apparatus provided by the embodiments of the present application. As shown in the figure, the communication apparatus 140 comprises a processing module 1401 and a transceiver module 1402. The transceiver module 1402 can realize corresponding communication functions, and the processing module 1401 is used for data processing. The transceiver module 1402 can also be referred to as an interface, a communication interface or a communication module, etc. Figure 4

[0293] In some embodiments of the present application, the communication apparatus can be used to perform the actions performed by the first device or the second device in the above method embodiments. The transceiver module 1402 is used to perform the transceiver-related operations performed by the first device or the second device in the above method embodiments, such as the sending end being the device itself or a chip or a functional module configurable in the device, etc. The processing module 1401 is used to perform the processing-related operations performed by the first device or the second device in the above method embodiments. The processing module 1401 can perform corresponding operations by calling a computer program, or can perform corresponding operations by a corresponding hardware circuit. The transceiver module 1402 can independently perform transceiver operations, or can perform corresponding transceiver operations under the control of the processing module 1401.

[0294] Exemplarily, Figure 12 As shown in the figure, the communication apparatus 140 can be the first device or a device in the first device in the above method embodiments. The processing module 1401 and the transceiver module 1402 in the communication apparatus 140 can perform the following operations respectively:

[0295] The transceiver module 1402 is used to obtain sampling capability information of the first device and / or the second device, wherein the sampling capability information is information used to represent the support capability of the sampling process, and the sampling process is a process in which the first device communicates with the second device to obtain sampling content;

[0296] The processing module 1401 is used to determine the type of the reinforcement learning algorithm according to the sampling capability information, wherein the type of the reinforcement learning algorithm comprises a heterogeneous policy type or a homogeneous policy type, the reinforcement learning algorithm defines the sampling process, and the reinforcement learning algorithm is used to realize the communication function.

[0297] In an optional implementation, the sampling content comprises an action parameter, a state parameter and a reward parameter. The action parameter is used to represent the characteristics of the communication operation performed by the first device on the second device in the sampling process. The state parameter is used to represent the loss of the communication operation. The reward parameter is used to represent the network performance corresponding to the communication operation.

[0298] ​In one alternative implementation, the sampling capability information includes one or more of the following: the sampling frequency supported by the device, sampling power consumption, storage capacity, air interface load, interference level, and transmission requirements.

[0299] In one alternative implementation, the communication function is power control, the communication operation is sending a communication signal with a first power, the action parameter is the first power, the state parameter is the path loss between the first device and the second device, and the reward parameter is the signal-to-interference-plus-noise ratio of the communication signal.

[0300] In one optional implementation, the communication function is link adaptation, the communication operation is sending a communication signal using a first modulation and coding scheme, the action parameter is the first modulation and coding scheme, the status parameters include the channel state information (CSI) between the first device and the second device and the historical modulation and coding scheme, and the reward parameter is the transport block size for transmitting the communication signal.

[0301] In one alternative implementation, the transceiver module 1402 is specifically used to receive sampling capability information of the second device in order to acquire the sampling capability of the first device and / or the second device.

[0302] In an optional implementation, the transceiver module 1402 is further configured to send a first request message to the second device, wherein the first request message is used to request sampling capability information.

[0303] In one alternative implementation, in determining the type of reinforcement learning algorithm based on sampling capability information, processing module 1401 is specifically configured to:

[0304] The type of reinforcement learning algorithm is determined based on the sampling capability information of the first and second devices.

[0305] In an optional implementation, the processing module 1401 is further configured to select a reinforcement learning algorithm according to the type of reinforcement learning algorithm, and execute the reinforcement learning algorithm.

[0306] In an optional implementation, the transceiver module 1402 is further configured to acquire sampling information from the third device via the second device.

[0307] In one alternative implementation, in determining the type of reinforcement learning algorithm based on sampling capability information, processing module 1401 is specifically configured to:

[0308] The type of reinforcement learning algorithm is determined based on the sampling capability information of the first, second, and third devices.

[0309] In one optional implementation, the first device is a base station and the second device is a user equipment;

[0310] Alternatively, the first device may be a user equipment and the second device may be a base station;

[0311] Alternatively, the first device and the second device may be two different base stations;

[0312] Alternatively, the first device may be a Radio Access Network (RIC) device, the second device may be a base station, and the third device may be a user equipment.

[0313] Reuse Figure 12 In other embodiments of this application, exemplarily, Figure 12 The communication device 140 shown can be the first device or a component in the first device described in the method embodiment above. The processing module 1401 and the transceiver module 1402 in the communication device 140 can respectively perform the following operations:

[0314] The first device receives type information sent by the second device, wherein the type information includes the type of reinforcement learning algorithm, the type of reinforcement learning algorithm includes different policy type or same policy type, the reinforcement learning algorithm is used to implement communication function, the reinforcement learning algorithm defines the sampling process, the sampling process is the process of the first device and the second device communicating to obtain sampling content, and the type information is information determined according to the sampling capability information, the sampling capability information is used to characterize the support capability for the sampling process;

[0315] Select a reinforcement learning algorithm based on its type and execute the reinforcement learning algorithm.

[0316] In one optional implementation, the sampling content includes action parameters, state parameters, and reward parameters. The action parameters are used to characterize the characteristics of the communication operation performed by the first device against the second device during the sampling process, the state parameters are used to characterize the loss of the communication operation, and the reward parameters are used to characterize the network performance corresponding to the communication operation.

[0317] In one alternative implementation, the sampling capability information includes one or more of the following: the sampling frequency supported by the device, sampling power consumption, storage capacity, air interface load, interference level, and transmission requirements.

[0318] In one alternative implementation, the communication function is power control, the communication operation is sending a communication signal with a first power, the action parameter is the first power, the state parameter is the path loss between the first device and the second device, and the reward parameter is the signal-to-interference-plus-noise ratio of the communication signal.

[0319] In one optional implementation, the communication function is link adaptation, the communication operation is sending a communication signal using a first modulation and coding scheme, the action parameter is the first modulation and coding scheme, the status parameters include the channel state information (CSI) between the first device and the second device and the historical modulation and coding scheme, and the reward parameter is the transport block size for transmitting the communication signal.

[0320] In an alternative implementation, the transceiver module 1402 is further configured to send the sampling capability information of the first device to the second device.

[0321] In an optional implementation, the transceiver module 1402 is further configured to receive a first indication message, wherein the first indication message is used to indicate the use of a reinforcement learning algorithm.

[0322] In an optional implementation, the processing module 1401 is further configured to select a reinforcement learning algorithm according to the type of reinforcement learning algorithm, and execute the reinforcement learning algorithm.

[0323] Reuse Figure 12 In other embodiments of this application, exemplarily, Figure 12 The communication device 140 shown can be the second device or a component in the second device described in the above method embodiment. The processing module 1401 and the transceiver module 1402 in the communication device 140 can respectively perform the following operations:

[0324] The processing module 1401 is used to determine the sampling capability information of the second device. The sampling capability information is used to characterize the ability to support the sampling process. The sampling process is the process of the first device and the second device communicating to obtain the sampling content. The sampling capability information is used to determine the type of reinforcement learning algorithm. The type of reinforcement learning algorithm includes heterogeneous or homogeneous strategies. The reinforcement learning algorithm defines the sampling process and is used to implement the communication function.

[0325] The transceiver module 1402 sends the sampling capability information of the second device to the first device.

[0326] In one optional implementation, the sampling content includes action parameters, state parameters, and reward parameters. The action parameters are used to characterize the characteristics of the communication operation performed by the first device against the second device during the sampling process, the state parameters are used to characterize the loss of the communication operation, and the reward parameters are used to characterize the network performance corresponding to the communication operation.

[0327] In one alternative implementation, the sampling capability information includes one or more of the following: the sampling frequency supported by the device, sampling power consumption, storage capacity, air interface load, interference level, and transmission requirements.

[0328] In one alternative implementation, the communication function is power control, the communication operation is sending a communication signal with a first power, the action parameter is the first power, the state parameter is the path loss between the first device and the second device, and the reward parameter is the signal-to-interference-plus-noise ratio of the communication signal.

[0329] In one optional implementation, the communication function is link adaptation, the communication operation is sending a communication signal using a first modulation and coding scheme, the action parameter is the first modulation and coding scheme, the status parameters include the channel state information (CSI) between the first device and the second device and the historical modulation and coding scheme, and the reward parameter is the transport block size for transmitting the communication signal.

[0330] In an optional implementation, the transceiver module 1402 is further configured to receive a first request message sent by the first device, the first request message being used to request sampling capability information.

[0331] In an alternative implementation, the transceiver module 1402 is further configured to send sampling capability information of the third device to the first device.

[0332] In one optional implementation, the first device is a base station and the second device is a user equipment;

[0333] Alternatively, the first device may be a user equipment and the second device may be a base station;

[0334] Alternatively, the first device and the second device may be two different base stations;

[0335] Alternatively, the first device may be a Radio Access Network (RIC) device, the second device may be a base station, and the third device may be a user equipment.

[0336] Reuse Figure 12 In other embodiments of this application, exemplarily, Figure 12 The communication device 140 shown can be the second device or a component in the second device described in the above method embodiment. The processing module 1401 and the transceiver module 1402 in the communication device 140 can respectively perform the following operations:

[0337] The transceiver module 1402 is used to acquire sampling capability information of the first device and / or the second device, wherein the sampling capability information is used to characterize the support capability for the sampling process, and the sampling process is the process of the first device and the second device communicating to obtain the sampling content;

[0338] The processing module 1401 is used to determine the type of reinforcement learning algorithm based on the sampling capability information. The type of reinforcement learning algorithm includes heterogeneous or homogeneous policy types. The reinforcement learning algorithm defines the sampling process and is used to implement communication functions.

[0339] In one optional implementation, the sampling content includes action parameters, state parameters, and reward parameters. The action parameters are used to characterize the characteristics of the communication operation performed by the first device against the second device during the sampling process, the state parameters are used to characterize the loss of the communication operation, and the reward parameters are used to characterize the network performance corresponding to the communication operation.

[0340] In one alternative implementation, the sampling capability information includes one or more of the following: the sampling frequency supported by the device, sampling power consumption, storage capacity, air interface load, interference level, and transmission requirements.

[0341] In one alternative implementation, the communication function is power control, the communication operation is sending a communication signal with a first power, the action parameter is the first power, the state parameter is the path loss between the first device and the second device, and the reward parameter is the signal-to-interference-plus-noise ratio of the communication signal.

[0342] In one optional implementation, the communication function is link adaptation, the communication operation is sending a communication signal using a first modulation and coding scheme, the action parameter is the first modulation and coding scheme, the status parameters include the channel state information (CSI) between the first device and the second device and the historical modulation and coding scheme, and the reward parameter is the transport block size for transmitting the communication signal.

[0343] In an alternative implementation, the transceiver module 1402 is further configured to receive sampling capability information of the first device.

[0344] In an optional implementation, the transceiver module 1402 is further configured to send type information to the first device, wherein the type information includes the type of reinforcement learning algorithm.

[0345] The specific descriptions of the transceiver module 1402 and the processing module 1401 shown in the above embodiments are merely examples. For the specific functions or execution steps of the transceiver module 1402 and the processing module 1401, please refer to the above method embodiments, which will not be described in detail here.

[0346] The communication device according to the embodiments of this application has been described above. The following describes possible product forms of the communication device. Any device possessing the above-described... Figure 12 Any product in any form that incorporates the functionality of a communication device falls within the protection scope of the embodiments of this application.

[0347] The following description is merely an example and does not limit the product form of the communication device in the embodiments of this application to this.

[0348] In one possible implementation, Figure 12In the communication device 140 shown, the processing module 1401 can be one or more processors, and the transceiver module 1402 can be a transceiver, or the transceiver module 1402 can also be a transmitting module and a receiving module. The transmitting module can be a transmitter, and the receiving module can be a receiver. The transmitting module and the receiving module are integrated into one device, such as a transceiver. In this embodiment, the processor and the transceiver can be coupled, etc., and the connection method between the processor and the transceiver is not limited in this embodiment. In the process of executing the above method, the process of sending information in the above method can be the process of the processor outputting the above information. When outputting the above information, the processor outputs the above information to the transceiver so that the transceiver can transmit it. After the above information is output by the processor, it may need to undergo other processing before reaching the transceiver. Similarly, the process of receiving information in the above method can be the process of the processor receiving the input above information. When the processor receives the input information, the transceiver receives the above information and inputs it into the processor. Furthermore, after the transceiver receives the above information, the above information may need to undergo other processing before being input into the processor.

[0349] like Figure 12 As shown, the communication device 150 includes one or more processors 1502 and transceivers 1501. Exemplarily, the transceiver 1501 is used to perform actions such as... Figure 12 The transceiver module 1402 shown implements the functions or steps, and the processor 1502 is used to execute such functions or steps. Figure 13 The processing module 1401 shown illustrates the functions or steps implemented by this module. For detailed information on the processor 1502 and transceiver 1501, please refer to [link / reference needed]. Figure 12 Alternatively, the method embodiments shown above will not be described in detail here.

[0350] The descriptions of the relevant steps and information in the above embodiments can be found in the descriptions of the method embodiments above, and will not be detailed here.

[0351] exist Figure 12 In various implementations of the communication apparatus shown, the transceiver may include a receiver for performing a receiving function (or operation) and a transmitter for performing a transmitting function (or operation). The transceiver is also used to communicate with other devices / appliances via a transmission medium.

[0352] Optionally, the communication device 150 may further include one or more memories 1503 for storing program instructions and / or data. The memory 1503 is coupled to the processor 1502. The coupling in this embodiment is an indirect coupling or communication connection between devices, units, or modules, and can be electrical, mechanical, or other forms, used for information exchange between devices, units, or modules. The processor 1502 may operate in conjunction with the memory 1503. The processor 1502 may execute program instructions stored in the memory 1503. Optionally, at least one of the above-mentioned memories may be included in the processor.

[0353] This application embodiment does not limit the specific connection medium between the transceiver 1501, processor 1502, and memory 1503. This application embodiment... Figure 12 The memory 1503, processor 1502, and transceiver 1501 are connected via a bus 1504. Figure 13 The connections between other components are shown in bold and are for illustrative purposes only, not as limiting information. The bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, Figure 13 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0354] In the embodiments of this application, the processor may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules within the processor.

[0355] In this application embodiment, the memory may include, but is not limited to, non-volatile memory such as hard disk drive (HDD) or solid-state drive (SSD), random access memory (RAM), erasable programmable read-only memory (EPROM), read-only memory (ROM), or compact disc read-only memory (CD-ROM), etc. Memory is any storage medium capable of carrying or storing program code in the form of instructions or data structures, and capable of being read and / or written by a computer (such as the communication device shown in this application), but is not limited to this. The memory in this application embodiment may also be a circuit or any other device capable of implementing storage functions, used to store program instructions and / or data.

[0356] Processor 1502 is primarily used for processing communication protocols and data, controlling the entire communication device, executing software programs, and processing software program data. Memory 1503 is primarily used for storing software programs and data. Transceiver 1501 may include control circuitry and an antenna. The control circuitry is primarily used for converting baseband signals to radio frequency signals and processing radio frequency signals. The antenna is primarily used for transmitting and receiving radio frequency signals in the form of electromagnetic waves. Input / output devices, such as touchscreens, displays, and keyboards, are primarily used for receiving user input data and outputting data to the user.

[0357] When the communication device is powered on, the processor 1502 can read the software program in the memory 1503, interpret and execute the instructions of the software program, and process the data of the software program. When data needs to be transmitted wirelessly, the processor 1502 performs baseband processing on the data to be transmitted and outputs the baseband signal to the radio frequency (RF) circuit. The RF circuit processes the baseband signal and transmits the RF signal outward in the form of electromagnetic waves through the antenna. When data is sent to the communication device, the RF circuit receives the RF signal through the antenna, converts the RF signal into a baseband signal, and outputs the baseband signal to the processor 1502. The processor 1502 converts the baseband signal into data and processes the data.

[0358] In another implementation, the radio frequency circuitry and antenna can be set up independently of the processor performing baseband processing. For example, in a distributed scenario, the radio frequency circuitry and antenna can be arranged remotely, independent of the communication device.

[0359] The communication device shown in the embodiments of this application may also have a higher... Figure 13This application does not limit the use of other components or other related elements. The methods performed by the processor and transceiver shown above are merely examples; the specific steps performed by the processor and transceiver can be found in the methods described above.

[0360] In another possible implementation Figure 13 In the communication device shown, the processing module 1401 can be one or more logic circuits, and the transceiver module 1402 can be an input / output interface, or a communication interface, or an interface circuit, or an interface, etc. Alternatively, the transceiver module 1402 can also be a transmitting module and a receiving module; the transmitting module can be an output interface, and the receiving module can be an input interface, integrated into one module, such as an input / output interface. Figure 13 As shown, Figure 12 The communication device 160 shown includes logic circuitry 1601 and interface 1602. That is, the processing module 1401 can be implemented using logic circuitry 1601, and the transceiver module 1402 can be implemented using interface 1602. The logic circuitry 1601 can be a chip, processing circuit, integrated circuit, or system-on-chip (SoC) chip, etc., and the interface 1602 can be a communication interface, input / output interface, pins, etc. For example, Figure 14 Taking the aforementioned communication device as an example, the chip includes a logic circuit 1601 and an interface 1602.

[0361] In this embodiment, the logic circuit and the interface can also be coupled to each other. The specific connection method between the logic circuit and the interface is not limited in this embodiment. For example, the logic circuit 1601 can be used to perform... Figure 14 The interface 1602 can be used to execute the functions or steps implemented by the processing module 1401 shown. Figure 14 The transceiver module 1402 shown illustrates the functions or steps implemented by this module. For detailed information on the logic circuit 1601 and interface 1602, please refer to [link / reference needed]. Figure 12 Alternatively, the method embodiments shown above will not be described in detail here.

[0362] The above description of the communication device is merely an example; for... Figure 12 For a detailed description of the communication device shown, please refer to the method embodiments above or Figure 13 or Figure 14 This will not be elaborated upon here.

[0363] The communication device shown in the embodiments of this application can implement the method provided in the embodiments of this application in hardware form, or it can implement the method provided in the embodiments of this application in software form, etc., and the embodiments of this application do not limit it in this way.

[0364] The descriptions of relevant steps and information in the above embodiments can be found in the descriptions of the method embodiments above, and will not be detailed here. For Figure 12 Figure 13 Figure 14 For specific implementations of the various embodiments shown, please refer to the above embodiments, which will not be described in detail here.

[0365] This application also provides a communication system, which includes a first device and a second device. The first device and the second device interact with each other. The first device is used to perform all or part of the operations of the first device in any of the foregoing method embodiments, and the second device is used to perform all or part of the operations of the second device in any of the foregoing method embodiments.

[0366] In addition, this application also provides a computer program for implementing the operations and / or processes performed by various computing resource management devices in the methods provided in this application.

[0367] This application also provides a computer-readable storage medium storing computer code that, when executed on a computer, causes the computer to perform the operations and / or processes performed by various computing resource management devices in the methods provided in this application.

[0368] This application also provides a computer program product comprising computer code or a computer program that, when run on a computer, causes the operations and / or processes performed by various entities in the method provided in this application to be executed.

[0369] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, or it may be an electrical, mechanical, or other form of connection.

[0370] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the technical effects of the solutions provided in the embodiments of this application.

[0371] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0372] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned readable storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0373] In this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0374] Furthermore, unless otherwise stated, the use of ordinal numbers such as "first" and "second" in the embodiments of this application is for distinguishing multiple objects, and is not for limiting the order, timing, priority or importance of multiple objects, such as the first request message and the first device.

[0375] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for selecting a reinforcement learning algorithm, characterized in that, The method includes: Obtain sampling capability information of a first device and / or a second device, wherein the sampling capability information is used to characterize the ability to support the sampling process, and the sampling process is the process by which the first device and the second device communicate to obtain sampling content; The type of reinforcement learning algorithm is determined based on the sampling capability information, wherein the type of reinforcement learning algorithm includes heterogeneous policy type or homogeneous policy type, the reinforcement learning algorithm defines the sampling process, and the reinforcement learning algorithm is used to implement communication function.

2. The method according to claim 1, characterized in that, The sampling content includes action parameters, state parameters, and reward parameters. The action parameters are used to characterize the characteristics of the communication operation performed by the first device against the second device during the sampling process. The state parameters are used to characterize the loss of the communication operation. The reward parameters are used to characterize the network performance corresponding to the communication operation.

3. The method according to claim 1 or 2, characterized in that, The sampling capability information includes one or more of the following: the sampling frequency supported by the device, sampling power consumption, storage capacity, air interface load, interference level, and transmission requirements.

4. The method according to any one of claims 1-3, characterized in that, The communication functions are one or more of the following: link adaptation, power control, interference coordination, load balancing, access control, and mobility management.

5. The method according to claim 4, characterized in that, The communication function is power control, the communication operation is sending a communication signal with a first power, the action parameter is the first power, the state parameter is the path loss between the first device and the second device, and the reward parameter is the signal-to-interference-plus-noise ratio of the communication signal.

6. The method according to claim 4, characterized in that, The communication function is link adaptive, the communication operation is to send a communication signal using a first modulation and coding scheme, the action parameter is the first modulation and coding scheme, the status parameter includes the channel state information (CSI) and historical modulation and coding scheme between the first device and the second device, and the reward parameter is the transport block size for transmitting the communication signal.

7. The method according to any one of claims 1-6, characterized in that, Applied to the first device, the acquisition of the sampling capabilities of the first device and / or the second device includes: Receive the sampling capability information of the second device.

8. The method according to claim 7, characterized in that, The method further includes: Send a first request message to the second device, wherein the first request message is used to request sampling capability information.

9. The method according to claim 7 or 8, characterized in that, The step of determining the type of reinforcement learning algorithm based on the sampling capability information includes: The type of reinforcement learning algorithm is determined based on the sampling capability information of the first device and the second device.

10. The method according to any one of claims 7-9, characterized in that, The method further includes: Receive a first instruction message, wherein the first instruction message is used to indicate the use of a reinforcement learning algorithm.

11. The method according to any one of claims 7-10, characterized in that, The method further includes: The reinforcement learning algorithm is selected according to its type, and then executed.

12. The method according to any one of claims 7-11, characterized in that, The method further includes: The sampling information of the third device is obtained through the second device.

13. The method according to claim 12, characterized in that, The step of determining the type of reinforcement learning algorithm based on the sampling capability information includes: The type of reinforcement learning algorithm is determined based on the sampling capability information of the first device, the second device, and the third device.

14. The method according to claims 1-13, characterized in that: The first device is a base station, and the second device is a user equipment. Alternatively, the first device may be a user equipment and the second device may be a base station; Alternatively, the first device and the second device may be two different base stations; Alternatively, the first device may be a Radio Access Network Intelligent Control (RIC) device, the second device may be a base station, and the third device may be a user equipment.

15. The method according to any one of claims 1-6, characterized in that, Applied to the second device, the acquisition of sampling capability information includes: Receive the sampling capability information of the first device.

16. The method according to claim 15, characterized in that, The step of determining the type of reinforcement learning algorithm based on the sampling capability information includes: The type of reinforcement learning algorithm is determined based on the sampling capability information of the first device and the sampling capability information of the second device.

17. The method according to claim 15 or 16, characterized in that, The method further includes: Type information is sent to the first device, wherein the type information includes the type of the reinforcement learning algorithm.

18. A method for selecting a reinforcement learning algorithm, characterized in that, Applied to a second device, the method includes: The sampling capability information of the second device is determined, wherein the sampling capability information is used to characterize the support capability for the sampling process, the sampling process is the process of the first device and the second device communicating to obtain sampling content, the sampling capability information is used to determine the type of reinforcement learning algorithm, the type of reinforcement learning algorithm includes hetero-policy type or homo-policy type, the reinforcement learning algorithm defines the sampling process, and the reinforcement learning algorithm is used to implement the communication function; Send the sampling capability information of the second device to the first device.

19. A method for selecting a reinforcement learning algorithm, characterized in that, Applied to a first device, the method includes: The first device receives type information sent by the second device, wherein the type information includes the type of reinforcement learning algorithm, the type of reinforcement learning algorithm includes different policy type or same policy type, the reinforcement learning algorithm is used to implement communication function, the reinforcement learning algorithm defines a sampling process, the sampling process is the process of the first device and the second device communicating to obtain sampling content, the type information is information determined according to the sampling capability information of the first device and / or the second device, and the sampling capability information is used to characterize the support capability for the sampling process; The reinforcement learning algorithm is selected according to its type, and then executed.

20. The method according to claim 19, characterized in that, The method further includes: Send the sampling capability information of the first device to the second device.

21. The method according to claim 19 or 20, characterized in that, The method further includes: Receive a first instruction message, wherein the first instruction message is used to indicate the use of a reinforcement learning algorithm.

22. A communication device, characterized in that, The communication device includes a module for performing the method as described in any one of claims 1-21; or, the communication device includes a processor configured to cause the communication device to implement the method as described in any one of claims 1-21.

23. A communication device, characterized in that, It includes logic circuitry and an interface, the interface being used for inputting and / or outputting information, and the logic circuitry being used to enable the communication device to implement the method as described in any one of claims 1-21.

24. A communication system, characterized in that, The communication system includes a first device and a second device, wherein the first device is configured to perform the method as described in any one of claims 1-14, and the second device is configured to perform the method as described in claim 18.

25. A communication system, characterized in that, The communication system includes a first device and a second device, wherein the first device is used to perform the method as described in any one of claims 19-21, and the second device is used to perform the method as described in any one of claims 1-6 and 15-17.

26. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program, which, when executed, performs the method as described in any one of claims 1-21.

Images