Mechanism for reinforcement learning for beam management

Abstract

The disclosure relates to reinforcement learning (RL) with respect to beam management. In particular, it utilizes sidelink capabilities to enable real data / traffic exchange for RL exploration training steps. In this way, on one hand it enables radio system performance-friendly RL learning training operations, and on the other hand will utilize available radio air resources.

Description

Technical Field

[0001] Various example embodiments of the present disclosure relate generally to the field of telecommunications, and in particular to methods, devices, apparatuses, and computer-readable storage media for reinforcement learning on beam management. Background Art

[0002] There are multiple examples of machine learning (ML) algorithms for wireless communication networks (e.g., next generation radio access networks (NG-RAN)) that provide various radio resource management (RRM) improvements. Machine learning, such as reinforcement learning or deep reinforcement learning, can be used as a decision-making tool to solve complex problems that are high-dimensional given the number of control parameters to be optimized and the strict timing constraints. Therefore, it is worthwhile to further investigate AI / ML techniques. Summary of the invention

[0003] In a first aspect of the present disclosure, a device is provided. The device includes at least one processor; and at least one memory, wherein the at least one memory stores instructions, and when the instructions are executed by the at least one processor, the device causes the device to: receive a continuous learning configuration from a network device, the continuous learning configuration indicating at least one of the following: a neighboring device, a first identifier of a first beam, or a second identifier of a second beam; transmit a side link message including data and a second identifier of a second beam to the neighboring device; and transmit data to the network device via the first beam.

[0004] In a second aspect of the present disclosure, a device is provided. The device includes at least one processor; and at least one memory, wherein the at least one memory stores instructions, and when the instructions are executed by the at least one processor, the device: receives a continuous learning configuration from a network device, the continuous learning configuration indicating at least one of the following: a first identifier of a device, a first beam, or a second identifier of a second beam; receives a side link message including data and a second identifier of a second beam from a neighboring device; and transmits data to the network device via the second beam.

[0005] In a third aspect of the present disclosure, a device is provided. The device includes at least one processor; and at least one memory, wherein the at least one memory stores instructions, and when the instructions are executed by the at least one processor, the device causes the device to: transmit a continuous learning configuration to a first terminal device, wherein the continuous learning configuration indicates at least one of the following: a second terminal device, a first identifier of a first beam, or a second identifier of a second beam; transmit the continuous learning configuration to a second terminal device; receive data from the first terminal device via the first beam; and receive data from the second terminal device via the second beam.

[0006] In a fourth aspect of the present disclosure, a method is provided. The method includes: receiving a continuous learning configuration from a network device, the continuous learning configuration indicating at least one of the following: a neighboring device, a first identifier of a first beam, or a second identifier of a second beam; transmitting a side link message including data and the second identifier of the second beam to the neighboring device; and transmitting data to the network device via the first beam.

[0007] In a fifth aspect of the present disclosure, a method is provided. The method includes: receiving a continuous learning configuration from a network device, the continuous learning configuration indicating at least one of the following: a device, a first identifier of a first beam, or a second identifier of a second beam; receiving a side link message including data and the second identifier of the second beam from a neighboring device; and transmitting data to the network device via the second beam.

[0008] In a sixth aspect of the present disclosure, a method is provided. The method includes: transmitting a continuous learning configuration to a first terminal device, the continuous learning configuration indicating at least one of the following: a second terminal device, a first identifier of a first beam, or a second identifier of a second beam; transmitting the continuous learning configuration to the second terminal device; receiving data from the first terminal device via the first beam; and receiving data from the second terminal device via the second beam.

[0009] In a seventh aspect of the present disclosure, a first device is provided. The first device includes a component for receiving a continuous learning configuration from a network device, the continuous learning configuration indicating at least one of the following: a neighboring device, a first identifier of a first beam, or a second identifier of a second beam; a component for transmitting a side link message including data and the second identifier of the second beam to the neighboring device; and a component for transmitting data to the network device via the first beam.

[0010] In an eighth aspect of the present disclosure, a second device is provided. The second device includes a component for receiving a continuous learning configuration from a network device, the continuous learning configuration indicating at least one of the following: a device, a first identifier of a first beam, or a second identifier of a second beam; a component for receiving a side link message including data and a second identifier of a second beam from a neighboring device; and a component for transmitting data to the network device via the second beam.

[0011] In a ninth aspect of the present disclosure, a third device is provided. The third device includes: a component for transmitting a continuous learning configuration to a first terminal device, the continuous learning configuration indicating at least one of the following: a second terminal device, a first identifier of a first beam, or a second identifier of a second beam; a component for transmitting the continuous learning configuration to a second terminal device; a component for receiving data from the first terminal device via the first beam; and a component for receiving data from the second terminal device via the second beam.

[0012] In a tenth aspect of the present disclosure, a computer-readable medium is provided, wherein the computer-readable medium includes instructions stored thereon, and the instructions are used to cause a device to at least execute the method according to the fourth aspect.

[0013] In an eleventh aspect of the present disclosure, a computer-readable medium is provided, wherein the computer-readable medium includes instructions stored thereon, and the instructions are used to cause a device to at least execute the method according to the fifth aspect.

[0014] In a twelfth aspect of the present disclosure, a computer-readable medium is provided, wherein the computer-readable medium includes instructions stored thereon, and the instructions are used to cause a device to at least execute the method according to the sixth aspect.

[0015] It should be understood that the invention summary is not intended to identify the key features or essential features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Some example embodiments will now be described with reference to the accompanying drawings, in which:

[0017] Figure 1 An example communication environment is shown in which example embodiments of the present disclosure may be implemented;

[0018] Figure 2 shows a schematic diagram of agent-environment interaction for reinforcement learning;

[0019] Figure 3 shows a signaling diagram of a lightweight testing mode for reinforcement learning according to an example embodiment of the present disclosure;

[0020] Figure 4 A schematic diagram showing an update model according to an example embodiment of the present disclosure is shown;

[0021] Figure 5 A flowchart showing a method implemented at a first device according to some example embodiments of the present disclosure is shown;

[0022] Figure 6 A flowchart showing a method implemented at a second device according to some example embodiments of the present disclosure is shown;

[0023] Figure 7 A flowchart of a method implemented at a third device according to some example embodiments of the present disclosure is shown;

[0024] Figure 8 shows a simplified block diagram of a device suitable for implementing an example embodiment of the present disclosure; and

[0025] Fig. 9A block diagram of an example computer-readable medium is shown according to some example embodiments of the present disclosure.

[0026] Throughout the drawings, the same or similar reference numerals refer to the same or similar elements. DETAILED DESCRIPTION

[0027] The principle of the present invention will now be described with reference to some example embodiments. It should be understood that these embodiments are described only for the purpose of illustration, and help those skilled in the art to understand and implement the present disclosure, without suggesting any limitation to the scope of the present disclosure. In addition to the embodiments described below, the embodiments described herein can be implemented in various ways.

[0028] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0029] References in this disclosure to "one embodiment," "an embodiment," "an example embodiment," etc. indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment necessarily includes the particular feature, structure, or characteristic. Moreover, such phrases do not necessarily refer to the same embodiment. Further, when a particular feature, structure, or characteristic is described in conjunction with an embodiment, it is considered to be within the knowledge of those skilled in the art to affect such feature, structure, or characteristic in conjunction with other embodiments, whether or not explicitly described.

[0030] It should be understood that, although the terms "first", "second" ... etc. in front of nouns in this article can be used to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element, and they do not limit the order of nouns. For example, without departing from the scope of the exemplary embodiment, the first element can be referred to as the second element, and similarly, the second element can be referred to as the first element. As used herein, the term "and / or" includes any and all combinations of one or more listed terms.

[0031] As used herein, “at least one of: ” and “at least one of ” and similar expressions, where a list of two or more elements is connected by “and” or “or”, refer to at least any one of the elements, or at least any two or more of the elements, or at least all of the elements.

[0032] As used herein, unless explicitly stated, performing a step “in response to A” does not indicate that the step is performed immediately after “A” occurs, and one or more intermediate steps may be involved.

[0033] The terms used herein are for the purpose of describing specific embodiments only and are not intended to limit the exemplary embodiments. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It is also to be understood that when used herein, the terms "comprises", "comprising", "has", "having", "includes" and / or "including" illustrate the presence of the features, elements and / or components, etc., but do not exclude the presence or addition of one or more other features, elements, components and / or combinations thereof.

[0034] As used in this application, the term "circuitry" may refer to one or more or all of the following: (a) hardware circuit implementation only (such as implementation only in analog and / or digital circuitry), and (b) a combination of hardware circuitry and software such as (where applicable): (i) a combination of analog and / or digital hardware circuitry and software / firmware, and (ii) any portion of a hardware processor with software (including a digital signal processor), software, and memory that work together to enable a device such as a mobile phone or server to perform various functions; and (c) A hardware circuit and / or processor, such as a microprocessor or portion of a microprocessor, that requires software (e.g., firmware) for operation, but the software may not be present when not needed for operation.

[0035] This definition of circuitry applies to all uses of the term in this application, including in any claims. As a further example, as used in this application, the term circuitry also covers an implementation of only a hardware circuit or a processor (or multiple processors) or a portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. For example and if applicable to a particular claim element, the term circuitry also covers a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in a server, cellular network device, or other computing or network device.

[0036] As used herein, the term "communication network" refers to a network that complies with any suitable communication standard, such as New Radio (NR), Long Term Evolution (LTE), Advanced LTE (LTE-A), Wideband Code Division Multiple Access (WCDMA), High Speed ​​Packet Access (HSPA), Narrowband Internet of Things (NB-IoT), etc. In addition, the communication between the terminal device and the network device in the communication network can be performed according to any suitable generation communication protocol, including but not limited to the first generation (1G), second generation (2G), 2.5G, 2.75G, third generation (3G), fourth generation (4G), 4.5G, fifth generation (5G), sixth generation (6G) communication protocol and / or any other protocol currently known or developed in the future. The embodiments of the present disclosure can be applied to various communication systems. In view of the rapid development of communication, there are of course future types of communication technologies and systems that can be used to implement the present disclosure. It should not be considered that the scope of the present disclosure is limited to the above-mentioned system.

[0037] As used herein, the term "network device" refers to a node in a communication network via which a terminal device accesses a network and receives services from it. A network device may refer to a base station (BS) or an access point (AP), such as a Node B (NodeB or NB), an evolved Node B (eNodeB or eNB), a NR NB (also referred to as a gNB), a remote radio unit (RRU), a radio head (RH), a remote radio head (RRH), a repeater, an integrated access and backhaul (IAB) node, a low-power node, such as a flying base station, a pico base station, a non-terrestrial network (NTN) or non-terrestrial network equipment, such as a satellite network device, a low earth orbit (LEO) satellite and a geosynchronous orbit (GEO) satellite, an aircraft network device, etc., depending on the terminology and technology applied. In some example embodiments, a radio access network (RAN) split architecture includes a centralized unit (CU) and a distributed unit (DU) at an IAB donor node. An IAB node includes a mobile terminal (IAB-MT) portion similar to a UE toward a parent node, and a DU portion of an IAB node is similar to a base station toward a next-hop IAB node.

[0038] The term "terminal device" refers to any terminal device capable of wireless communication. As an example and not limitation, a terminal device may also be referred to as a communication device, a user equipment (UE), a subscriber station (SS), a portable subscriber station, a mobile station (MS), or an access terminal (AT). The terminal device may include, but is not limited to, a mobile phone, a cellular phone, a smart phone, a voice over IP (VoIP) phone, a wireless local ring phone, a tablet computer, a wearable terminal device, a personal digital assistant (PDA), a portable computer, a desktop computer, an image capture terminal device such as a digital camera, a game terminal device, a music storage and playback device, a vehicle-mounted wireless terminal device, a wireless endpoint, a mobile station, a laptop embedded device (LEE), a laptop mounted device (LME), a USB dongle, a smart device, a wireless customer premises equipment (CPE), an Internet of Things (IoT) device, a watch or other wearable device, a head mounted display (HMD), a vehicle, a drone, medical equipment and applications (e.g., remote surgery), industrial equipment and applications (e.g., robots and / or other wireless devices operating in the context of an industrial and / or automated processing chain), consumer electronic devices, equipment operating on a commercial and / or industrial wireless network, etc. The terminal device may also correspond to the mobile terminal (MT) part of the IAB node (eg, relay node). In the following description, the terms "terminal device", "communication device", "terminal", "user equipment" and "UE" may be used interchangeably.

[0039] As used herein, the terms "resource", "transmission resource", "resource block", "physical resource block" (PRB), "uplink resource" or "downlink resource" may refer to any resource used to perform communication, for example, communication between a terminal device and a network device, such as time domain resources, frequency domain resources, spatial domain resources, code domain resources, or any other combination of time domain, frequency domain, spatial domain and / or code domain resources that enable communication. In the following, unless explicitly stated, resources in the frequency domain and time domain will be used as examples of transmission resources for describing some example embodiments of the present disclosure. Note that the example embodiments of the present disclosure are equally applicable to other resources in other domains. The term "model" used herein may refer to a data-driven algorithm that applies artificial intelligence (AI) / machine learning (ML) techniques to generate a set of outputs based on a set of inputs. The term "beam" used herein may refer to an index of a channel state information reference signal (CSI-RS) or an index of a synchronization signal physical broadcast channel (PBCH) signal (SSB). The term "continuous learning" used herein may refer to the ability of a model to continuously learn from a data stream. The term "reinforcement learning (RL)" as used herein may refer to a type of machine learning technique that enables an agent to learn by trial and error in an interactive environment using feedback from its own actions and experiences. Reinforcement learning may be a specific type of continuous learning. Typically, an RL agent is able to perceive and interpret its environment, take actions, and learn by trial and error. The term "reinforcement learning exploration" as used herein may refer to an essential component of a reinforcement learning algorithm, in which the agent needs to learn how to predict and control an unknown and often random environment. The term "reinforcement learning model" as used herein may refer to any model that will learn by trial and error. A "reinforcement learning model" may be a model based on a deep neural network or any statistical model.

[0040] As described above, machine learning such as reinforcement learning or deep reinforcement learning can be used as a decision-making tool to solve complex problems, which are high-dimensional given the number of control parameters to be optimized and the strict timing constraints. Reinforcement learning (RL) is a field of machine learning in which an agent uses a trial-and-error strategy to maximize a given performance goal. The agent chooses an action based on the current state of the environment. The action affects the environment and results in a new state and reward (the goodness of the action in the given state). The goal of the agent is to learn which actions with a given state result in the highest cumulative future reward.

[0041] The goal of trial and error is to gain knowledge about what works and what doesn't. This encompasses the concepts of exploration and exploitation, where exploration refers to choosing random actions, and exploitation means that the agent exploits the learned policy (i.e., the information received so far). In practice, RL algorithms must strike a balance between exploration and exploitation. Typically, RL algorithms start with a high probability of exploration before converging to a good policy, and reduce the probability of exploration as the agent begins to learn. An agent is usually considered to be learning when the reward received increases over time and stabilizes to a certain level.

[0042] Further, it is difficult to maintain good system performance during the RL-based model training process. This is mainly due to the importance of the exploration step (occasional random actions), which will have a direct impact on system performance due to taking less optimal actions when searching for the optimal action. Initially, the RL model will have poor performance, and directly using the poor model for making decisions will have a significant impact on system performance. Taking beam management (BM) based on RL QoS model as an example, this RL model will be trained to take actions (select the optimal beam) in order to maximize QoS, and therefore, for each action taken, the model receives a QoS level as a reward, and the goal is to maximize this reward by selecting the optimal beam. Assuming this, it requires the RL model to be trained with a QoS level that matches the QoS level when real user data / traffic is transmitted through the selected beam. To this end, it is necessary to transmit actual real data / traffic using different actions (different beams), and then evaluate the different achieved QoS levels for which each action is rewarded. In this case, the use of virtual synthetic data may work / may not work because the QoS level achieved using virtual data may be different from the QoS using real traffic. Besides this, using virtual data for training will waste radio resources to transmit data used only for RL model training. The term “virtual” here may refer to data that does not reflect the nature and distribution of real user data. However, using user actual data / traffic in the exploration step will lead to user dissatisfaction due to potential degradation of QoS level when suboptimal actions are taken. Therefore, solutions on how to use real data / traffic to train and evaluate RL models with minimal impact on actual user / system QoS and how to optimally utilize available radio resources during the RL training process are required.

[0043] According to an embodiment of the present disclosure, it utilizes the side link capability to achieve real data / traffic exchange for RL exploration training steps. In this way, it can achieve radio system performance-friendly RL learning training operations from one side, and will utilize available radio air resources on the other side. Typically, the network device trains the RL model, and then the network device provides input to the RL model and requests its action / output, that is, the beam identification (ID). In this case, the network device does not know whether the beam ID is currently the best or the action of the RL model. By allowing the RL model to take several actions for the same input, and then by sending the same primary user data on those beams, the chance that the data will be received by the utilized beam at the network device is increased, even if the data may not come from the primary user terminal and other exploration beams are tested at the same time. Further, the training time of the RL model can also be reduced with minimal impact on the system.

[0044] Figure 1 An example communication environment 100 is shown in which example embodiments of the present disclosure may be implemented. Communication environment 100 includes devices 110-1, 110-2, 110-3 (collectively referred to as devices 110), and 120 that can communicate with each other. Devices 110-1 and 110-2 may be proximate to a user with sidelink capabilities.

[0045] In the following, for the purpose of illustration, some example embodiments are described in which devices 110-1, 110-2, and 110-3 operate as terminal devices and device 120 operates as a network device. However, in some example embodiments, the operations described in conjunction with the terminal devices may be implemented at the network devices or other devices, and the operations described in conjunction with the network devices may be implemented at the terminal devices or other devices.

[0046] In some example embodiments, if device 110 is a terminal device and device 120 is a network device, the link from device 120 to device 110 is referred to as a downlink (DL), and the link from device 110 to device 120 is referred to as an uplink (UL). In the DL, device 120 is a transmitting (TX) device (or transmitter), and device 110 is a receiving (RX) device (or receiver). In the UL, device 110 is a TX device (or transmitter), and device 120 is an RX device (or receiver). The link between devices 110-1, 110-2, and 110-3 is referred to as a side link (SL).

[0047] AI / ML technology can be applied in the communication environment 100. AI / ML technology can be applied to a variety of scenarios. For example, the use case set of AI / ML technology can include channel state information (CSI) feedback enhancement, beam management (BM), and positioning accuracy enhancement. Figure 2Schematic diagram of agent-environment interaction for reinforcement learning that can be applied in a communication environment 100 is shown. Taking BM as an example, it is a key use case studied under the AI / ML topic, where supervised learning methods are explored to reduce the measurement overhead of BM, such as Figure 2 As shown in , RL-based approaches will be studied for BM, where the RL agent 210 selects beams to maximize throughput or minimize latency (QoS). Since the strongest beam in every situation / configuration (e.g., frequency range 2 (FR2) analog beamforming) does not always correlate to the best QoS, the RL agent 210 is trained to select DL beams to improve QoS. Here, the RL agent 210 needs to explore the environment 220 by applying different DL beams and observing the rewards, which can be throughput or latency (depending on the UE's QoS class).

[0048] The communication in the communication environment 100 may be implemented according to any suitable (multiple) communication protocols, including but not limited to cellular communication protocols of the first generation (1G), second generation (2G), third generation (3G), fourth generation (4G), fifth generation (5G), sixth generation (6G), etc., wireless local area network communication protocols such as Institute of Electrical and Electronics Engineers (IEEE) 802.11, etc., and / or any other protocol currently known or developed in the future. In addition, the communication may use any suitable wireless communication technology, including but not limited to: code division multiple access (CDMA), frequency division multiple access (FDMA), time division multiple access (TDMA), frequency division duplex (FDD), time division duplex (TDD), multiple input multiple output (MIMO), orthogonal frequency division multiple access (OFDM), discrete Fourier transform spread OFDM (DFT-s-OFDM) and / or any other technology currently known or developed in the future.

[0049] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0050] Reference now Figure 3 , which shows a signaling flow 300 for RL on BM according to some example embodiments of the present disclosure. For the purpose of discussion, reference will be made to Figure 1 Signaling flow 300 is discussed, for example, using device 110 - 1 , device 110 - 2 , device 110 - 3 , and device 120 .

[0051] Device 120 may transmit (3010) a request for side link measurements to device 110-1. In other words, device 110-1 may receive a request for side link measurements. In some example embodiments, the request may indicate a list of devices to be measured. The list of devices may include one or more idle terminal devices that are candidates for supporting RL exploration. That is, one or more idle terminal devices may be candidates for operating a lightweight test mode. The term "idle terminal device" used herein may refer to a terminal device that does not have data or information to transmit. In some example embodiments, the side link measurements may include reference signal received power (RSRP) measurements. Alternatively, the side link measurements may include positioning measurements. It should be noted that the side link measurements may include any appropriate type of measurement.

[0052] After receiving the request for sidelink measurement, device 110-1 may perform sidelink measurement with at least one neighboring device. For example, device 110-1 may perform (3020) sidelink measurement with device 110-2. Device 110-1 may also perform (3030) sidelink measurement with device 110-3. For example, upon receiving the request for sidelink measurement, device 110-1 may begin performing sidelink measurement to detect its neighboring devices (e.g., devices 110-2 and / or 110-3). By way of example, device 110-1 may measure RSRP from devices 110-2 and / or 110-3.

[0053] Device 110-1 may transmit (3040) the result of the sidelink measurement to device 120. In other words, device 120 may receive the result of the sidelink measurement from device 110-1. For example, device 110-1 may report the sidelink measurement of device 110-2 and device 110-3 to device 120. By way of example, if the sidelink measurement is an RSRP measurement, the measured (3020, 3030) RSRP of device 110-2 and device 110-3 may be reported to device 120.

[0054] Device 120 may determine a target device from at least one neighbor based on the result of the side link measurement received from device 110-1. For example, device 120 may analyze (3050) the result of the received side link measurement. In this case, device 120 may determine the target device from at least one neighboring device by analyzing the result of the received side link measurement. By way of example, device 120 may begin grouping devices into pairs for RL exploratory collaborative operations based on the result of the side link measurement. As an example, if the RSRP of device 110-2 is higher than the RSRP of device 110-3, device 120 may treat device 110-1 and device 110-2 as a pair for RL exploratory collaborative operations. In other words, device 120 may assign device 110-2 to lightweight test mode.

[0055] In some example embodiments, device 120 may acquire information related to data to be transmitted by device 110-1.Alternatively or additionally, device 120 may acquire first terminal-side link information related to device 110-2.

[0056] Device 120 may select a beam (hereinafter referred to as the "first beam") by a reinforcement learning model at device 120. According to the reinforcement learning model (i.e., the latest RL training model), the first beam may be selected as the optimal beam for transmission of equipment 110-1. Equipment 120 may select another beam (hereinafter referred to as the "second beam") for exploring the reinforcement learning model. According to the reinforcement learning model, the second beam may not be the optimal beam, but is selected for transmission as part of the RL exploration process. Since the transmission by device 110-2 is only used for RL purposes, the worst transmission with low QoS may be beneficial for RL updates, and the best QoS of the transmission may be high, which not only allows the RL to be updated accordingly, but also allows the transmitted data to be used on the gNB side. In this case, useful data can be transmitted in the uplink by two different sources.

[0057] In some example embodiments, device 120 may select multiple second beams for exploration of the reinforcement learning model. For example, device 120 may determine multiple neighboring devices of device 110-1, and each neighboring device in the multiple neighboring devices corresponds to one second beam in the multiple second beams.

[0058] Device 120 transmits (3060) a continuous learning configuration to device 110-1. In other words, device 110-1 receives the continuous learning configuration from device 120. In some example embodiments, the continuous learning configuration may indicate a neighboring device selected from at least one neighboring device. For example, if device 120 selects device 110-2, the continuous learning configuration may include an identifier of device 110-2. The identifier may be an identifier of any suitable type. Alternatively or additionally, the continuous learning configuration may indicate a first beam. For example, the continuous learning configuration may include an identifier (also referred to as a "first identifier") or an index of the first beam. The continuous learning configuration may also indicate a second beam. For example, the continuous learning configuration may include an identifier (also referred to as a "second identifier") or an index of the second beam. In some other example embodiments, the continuous learning configuration may further indicate a data type of data transmitted to device 120 based on the reinforcement learning model. For example, the continuous learning configuration may include a traffic type of the data. As described above, multiple neighboring devices and corresponding multiple second beams may be selected for exploring the reinforcement learning model. In this case, the continuous learning configuration may indicate the identities of the plurality of neighboring devices and the beam identities of the plurality of second beams.

[0059] Device 120 transmits (3070) a continuous learning configuration to at least one neighboring device (e.g., device 110-2). In other words, device 110-2 receives the continuous learning configuration from device 120. As mentioned above, the continuous learning configuration may indicate at least one of the following: a neighboring device (e.g., device 110-2), a first identification of the first beam, or a second identification of the second beam.

[0060] Device 110-1 transmits (3080) a side link message to device 110-2. In other words, device 110-2 receives a side link message from device 110-1. For example, device 110-2 is indicated in a continuous learning configuration. The side link message includes data to be transmitted to device 120. The side link message may also include a second identifier of the second beam. As described above, multiple neighboring devices and corresponding multiple second beams may be selected for exploration of the reinforcement learning model. In this case, device 110-1 may also transmit the side link message to multiple neighboring devices.

[0061] Device 110-1 transmits (3090) data to device 120 via the first beam. In other words, device 120 receives data from device 110-1 via the first beam. Device 110-2 transmits (3100) data to device 120 via the second beam. In other words, device 120 also receives data from device 110-2. As described above, multiple neighboring devices and corresponding multiple second beams can be selected for exploration of the reinforcement learning model. In this case, multiple neighboring devices can transmit data to device 120 via their corresponding beams. In other words, device 120 can receive data from multiple neighboring devices using different beams.

[0062] Device 120 may update ( 3110 ) a reinforcement learning model based on the data received ( 3090 ) from device 110 - 1 and the data received ( 3100 ) from device 110 - 2 . Figure 4 Different steps are shown for an example embodiment of an RL agent 210, which may be located at device 120 or any other entity of the network, and distinguishing between steps related to conventional data transmission and novel steps dedicated to a lightweight testing mode with a new transmission intended to enhance RL exploration. As described above, a plurality of neighboring devices and a corresponding plurality of second beams may be selected for exploration of a reinforcement learning model. In this case, the reinforcement learning model may be updated based on data received from the plurality of neighboring devices.

[0063] RL agent 210 may collect (401) information about different devices, where device 110-1 refers to an active UE with real data transmission, while other UEs (e.g., device 110-2) are idle and may be candidates for operating in a lightweight test mode. In step 401, RL agent 210 may collect information related to data to be transmitted by device 110-1 in the UL and sidelink information of neighboring candidate devices. Then, in step 402, beams for transmissions of devices 110-1 and 110-2 are selected by RL agent 210. In this case, device 110-1 is assigned an utilization beam, which refers to the best beam identified with the ongoing RL model. However, device 110-2 is assigned an exploration beam (because poor transmissions will not actually degrade real transmissions).

[0064] In step 403, RL agent 210 may receive reward information about both the transmission of device 110-1 and the transmission of device 110-2 and update its RL model accordingly. In some solutions, the decision about the transmission of device 110-1 is implemented in step 404 and is implemented after deciding the beam identified as the best beam according to the latest RL trained model. In this example, in step 405, device 110-2 is assigned to device 110-1 as the utilized beam id. Thereafter, the performance of the transmission (e.g., throughput / QoS KPI) is evaluated and shared with RL agent 210 in step 406. The embodiments of the present disclosure reply to the use of sidelink-based testing and exploration processes and lightweight test mode. In fact, in order to implement this RL exploration solution, it is first necessary to explore neighboring devices through dedicated sidelink measurements (standardized measurements) in step 407. Device 110-2 is thereafter identified as a candidate device for operating in lightweight test mode: meaning that it is close enough to device 110-1 and can provide assistance in the RL exploration process regardless of the impact of possible transmission failures. The data is then shared by device 110-1 to device 110-2 via the side link at step 408, and thereafter device 110-2 may perform (409) transmission of the data using (410) the beam id selected by RL agent 210. At step 411, the performance of device 110-2 using the second beam is evaluated, and the information is shared with RL agent 210.

[0065] Embodiments of the present disclosure propose signaling enhancements to initiate and enable lightweight test modes for RL exploration enhancement purposes. Using a sidelink to reuse real data for RL exploration training steps can allow real data to be used and used to train RL models to learn real goals. In addition to RL purposes, the same data of a device relayed by another device can have a good performance impact on the QoS of that device. In addition, this can compensate for the impact of RL training on radio air resources used for training purposes.

[0066] Figure 5 FIG. 5 is a flow chart showing an example method 500 implemented at a device according to some example embodiments of the present disclosure. For discussion purposes, the method 500 will be described in detail in FIG. Figure 1 The perspective of device 110 - 1 is described.

[0067] At block 510, device 110-1 receives a continuous learning configuration from a network device, the continuous learning configuration indicating at least one of: a neighboring device, a first identification of a first beam, or a second identification of a second beam. In some example embodiments, the first beam is acquired based on a reinforcement learning model at the network device, and wherein the second beam is selected for reinforcement learning exploration of the reinforcement learning model. In some example embodiments, the configuration further indicates a data type of data transmitted to the network device based on the reinforcement learning model. In some example embodiments, at least one of the first beam or the second beam is used for exploration of the reinforcement learning model.

[0068] At block 520, device 110-1 transmits a sidelink message including data and a second identification of a second beam to a neighboring apparatus.

[0069] At block 530, device 110-1 transmits data to the network device via the first beam. In some example embodiments, method 500 further includes: receiving a request for sidelink measurement from the network device; performing sidelink measurement with at least one neighboring device; and transmitting results of the sidelink measurement regarding the at least one neighboring device to the network device.

[0070] In some example embodiments, the apparatus comprises a terminal device and the neighboring apparatus comprises another terminal device.

[0071] Figure 6 FIG. 6 is a flow chart of an example method 600 implemented at a device according to some example embodiments of the present disclosure. For discussion purposes, the method 600 will be described in detail in FIG. Figure 1 1 and 12. The description is made from the perspective of device 110-2 in FIG.

[0072] At block 610, device 110-2 receives a continuous learning configuration from a network device, the continuous learning configuration indicating at least one of: an apparatus, a first identification of a first beam, or a second identification of a second beam. In some example embodiments, the first beam is acquired based on a reinforcement learning model at the network device, and wherein the second beam is selected for exploration of the reinforcement learning model. In some example embodiments, at least one of the first beam or the second beam is used for exploration of the reinforcement learning model.

[0073] At block 620, device 110-2 receives a sidelink message including data and a second identification of a second beam from a neighboring apparatus.

[0074] At block 630, device 110-2 transmits data to the network device via the second beam.

[0075] In some example embodiments, the configuration further indicates a data type of data transmitted to the network device based on the reinforcement learning model.

[0076] Figure 7 FIG. 7 is a flow chart showing an example method 700 implemented at a device according to some example embodiments of the present disclosure. For discussion purposes, the method 700 will be described in detail in FIG. Figure 1 The angle of the device 120 is described.

[0077] At block 710, device 120 transmits a continuous learning configuration to a first terminal device, the continuous learning configuration indicating at least one of: a second terminal device, a first identifier of a first beam, or a second identifier of a second beam. At block 720, device 120 transmits a continuous learning configuration to a second terminal device.

[0078] At block 730, device 120 receives data from a first terminal device via a first beam. At block 740, device 120 receives data from a second terminal device via a second beam.

[0079] In some example embodiments, method 700 further includes: transmitting a request for sidelink measurement to the first terminal device; receiving from the first terminal device a result of the sidelink measurement of at least one neighboring terminal device of the first terminal device; and determining a second terminal device from the at least one neighboring terminal device based on the result of the sidelink measurement received from the first terminal device.

[0080] In some example embodiments, method 700 further includes: acquiring information related to data to be transmitted by the first terminal device; and acquiring first terminal side link information related to the second terminal device.

[0081] In some example embodiments, the method 700 further includes: selecting, by the reinforcement learning model at the device, the first beam; and selecting, by the reinforcement learning model at the device, a second beam for exploration of the reinforcement learning model.

[0082] In some example embodiments, method 700 further includes: updating a reinforcement learning model at the apparatus based on the data received from the first terminal device and the data received from the second terminal device.

[0083] In some example embodiments, the apparatus is a network device.

[0084] In some example embodiments, a first device (eg, Figure 1 The device 110-1 in the method 500 may include a component for performing the corresponding operation of the method 500. The component may be implemented in any suitable form. For example, the component may be implemented in a circuit or a software module. The first device may be implemented as or included in Figure 1 In device 110-1.

[0085] In some example embodiments, the first device includes a component for receiving a continuous learning configuration from a network device, the continuous learning configuration indicating at least one of: a neighboring device, a first identifier of a first beam, or a second identifier of a second beam; a component for transmitting a side link message including data and the second identifier of the second beam to the neighboring device; and a component for transmitting data to the network device via the first beam.

[0086] In some example embodiments, the first beam is acquired based on a reinforcement learning model at the network device, and wherein the second beam is selected for exploration of the model.

[0087] In some example embodiments, at least one of the first beam or the second beam is used for exploration of a reinforcement learning model.

[0088] In some example embodiments, the configuration further indicates a data type of data transmitted to the network device based on the reinforcement learning model.

[0089] In some example embodiments, the first device also includes: a component for receiving a request for sidelink measurement from a network device; a component for performing sidelink measurement with at least one neighboring device; and a component for transmitting results of the sidelink measurement regarding the at least one neighboring device to the network device.

[0090] In some example embodiments, the apparatus comprises a terminal device and the neighboring apparatus comprises another terminal device.

[0091] In some example embodiments, the first apparatus further comprises means for performing method 500 or other operations of some example embodiments of device 110-1. In some example embodiments, the means comprises at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause execution of the first apparatus.

[0092] In some example embodiments, a second device (eg, Figure 1 The device 110-2 in the second device may include a component for performing the corresponding operation of the method 600. The component may be implemented in any suitable form. For example, the component may be implemented in a circuit or a software module. The second device may be implemented as Figure 1 The device 110 - 2 in or included in the device 110 - 2.

[0093] In some example embodiments, the second device includes a component for receiving a continuous learning configuration from the network device, the continuous learning configuration indicating at least one of the following: the device, the first identifier of the first beam, or the second identifier of the second beam; a component for receiving a side link message including data and the second identifier of the second beam from a neighboring device; and a component for transmitting data to the network device via the second beam.

[0094] In some example embodiments, the first beam is acquired based on a reinforcement learning model at the network device, and wherein the second beam is selected for exploration of the reinforcement learning model.

[0095] In some example embodiments, at least one of the first beam or the second beam is used for exploration of a reinforcement learning model.

[0096] In some example embodiments, the configuration further indicates a data type of data transmitted to the network device based on the reinforcement learning model.

[0097] In some example embodiments, the apparatus comprises a terminal device and the neighboring apparatus comprises another terminal device.

[0098] In some example embodiments, the second device further comprises means for performing method 600 or other operations in some example embodiments of device 110-2. In some example embodiments, the means comprises at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause execution of the second device.

[0099] In some example embodiments, a third device (eg, Figure 1The device 120 in the embodiment may include a component for performing the corresponding operation of the method 700. The component may be implemented in any suitable form. For example, the component may be implemented in a circuit or a software module. The third device may be implemented as Figure 1 The device 120 or is included in Figure 1 In the device 120.

[0100] In some example embodiments, the third apparatus includes: a component for transmitting a continuous learning configuration to the first terminal device, the continuous learning configuration indicating at least one of: a second terminal device, a first identifier of the first beam, or a second identifier of the second beam; a component for transmitting the continuous learning configuration to the second terminal device; a component for receiving data from the first terminal device via the first beam; and a component for receiving data from the second terminal device via the second beam.

[0101] In some example embodiments, the third apparatus further includes: a component for transmitting a request for sidelink measurement to the first terminal device; a component for receiving from the first terminal device a result of sidelink measurement of at least one neighboring terminal device of the first terminal device; and a component for determining the second terminal device from the at least one neighboring terminal device based on the result of the sidelink measurement received from the first terminal device.

[0102] In some example embodiments, the third apparatus further includes: a component for acquiring information related to data to be transmitted by the first terminal device; and a component for acquiring first terminal side link information related to the second terminal device.

[0103] In some example embodiments, the third device further includes: means for selecting the first beam by the reinforcement learning model at the device; and means for selecting, by the reinforcement learning model at the device, the second beam for exploration of the reinforcement learning model.

[0104] In some example embodiments, the third apparatus further comprises: means for updating a reinforcement learning model at the apparatus based on data received from the first terminal device and data received from the second terminal device.

[0105] In some example embodiments, the apparatus is a network device.

[0106] In some example embodiments, the third apparatus further comprises means for performing other operations in some example embodiments of method 700 or apparatus 120. In some example embodiments, the means comprises at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause execution of the third device.

[0107] Figure 88 is a simplified block diagram of a device 800 suitable for implementing an example embodiment of the present disclosure. The device 800 may be used to implement a communication device, such as Figure 1 The device 110 - 1 , the device 110 - 2 , or the device 120 are shown. As shown, the device 800 includes one or more processors 810 , one or more memories 820 coupled to the processors 810 , and one or more communication modules 840 coupled to the processors 810 .

[0108] The communication module 840 is used for two-way communication. The communication module 840 has one or more communication interfaces to facilitate communication with one or more other modules or devices. The communication interface may represent any interface necessary for communication with other network elements. In some example embodiments, the communication module 840 may include at least one antenna.

[0109] Processor 810 may be of any type suitable for the local technology network and may include one or more of the following: a general purpose computer, a special purpose computer, a microprocessor, a digital signal processor (DSP), and a processor based on a multi-core processor architecture, as non-limiting examples. Device 800 may have multiple processors, such as application specific integrated circuit chips, which are time-slaved to a clock that synchronizes a main processor.

[0110] The memory 820 may include one or more non-volatile memories and one or more volatile memories. Examples of non-volatile memories include, but are not limited to, read-only memory (ROM) 824, electrically programmable read-only memory (EPROM), flash memory, hard disk, compact disk (CD), digital video disk (DVD), optical disk, laser disk, and other magnetic and / or optical memories. Examples of volatile memories include, but are not limited to, random access memory (RAM) 822 and other volatile memories that will not survive the duration of a power outage.

[0111] The computer program 830 includes computer executable instructions executed by the associated processor 810. The instructions of the program 830 may include instructions for performing the operations / actions of some example embodiments of the present disclosure. The program 830 may be stored in a memory (e.g., ROM 824). The processor 810 may perform any suitable actions and processes by loading the program 830 into the RAM 822.

[0112] The exemplary embodiments of the present disclosure may be implemented by the program 830 so that the device 800 may execute the procedures described in the reference Figures 2 to 7 Any process of the present disclosure discussed. Example embodiments of the present disclosure may also be implemented by hardware or a combination of software and hardware.

[0113] In some example embodiments, the program 830 may be tangibly contained in a computer-readable medium, which may be contained in the device 800 (such as in the memory 820) or in other storage devices accessible by the device 800. The device 800 may load the program 830 from the computer-readable medium to the RAM 822 for execution. In some example embodiments, the computer-readable medium may include any type of non-transitory storage medium, such as ROM, EPROM, flash memory, hard disk, CD, DVD, etc. The term "non-transitory" as used herein is a limitation of the medium itself (i.e., tangible, not a signal), not a limitation on the persistence of data storage (e.g., RAM vs. ROM).

[0114] Fig. 9 An example of a computer readable medium 900 is shown, which may be in the form of a CD, DVD or other optical storage disk. The computer readable medium 900 has a program 830 stored thereon.

[0115] Generally, various embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects may be implemented in hardware, and other aspects may be implemented in firmware or software, which may be executed by a controller, microprocessor, or other computing device. Although various aspects of the embodiments of the present disclosure are illustrated and described as block diagrams, flow charts, or using some other graphical representations, it should be understood that, as non-limiting examples, the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general hardware or controllers or other computing devices, or some combination thereof.

[0116] Some example embodiments of the present disclosure also provide at least one computer program product tangibly stored on a computer-readable medium (such as, a non-transitory computer-readable medium). The computer program product includes computer executable instructions, such as those computer executable instructions that are executed in a device on a target physical or virtual processor included in a program module to implement any method as described above. Typically, a program module includes routines, programs, libraries, objects, classes, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The functionality of the program modules can be combined or split between program modules as needed in various embodiments. Machine executable instructions for program modules can be executed in local or distributed devices. In distributed devices, program modules can be located in local and remote storage media.

[0117] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller in a general-purpose computer, a special-purpose computer or other programmable data processing device so that the program code enables the functions / operations specified in the flow chart and / or block diagram to be implemented when executed by the processor or controller. The program code can be executed entirely on the machine, partially on the machine, as an independent software package, partially on the machine, partially on a remote machine, or entirely on a remote machine or server.

[0118] In the context of the present disclosure, computer program codes or related data may be carried by any suitable carrier to enable a device, apparatus or processor to perform the various processes and operations described above. Examples of carriers include signals, computer readable media, etc.

[0119] The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable medium may include but is not limited to electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. More specific examples of computer readable storage media will include an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0120] Further, although the operations are depicted in a specific order, this should not be understood as these operations need to be performed in the specific order shown or in a sequential order, or all the operations shown are to be performed to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be interpreted as limitations on the scope of the present disclosure, but should be interpreted as descriptions of features that may be specific to a specific embodiment. Unless explicitly stated, certain features described in the context of a separate embodiment may also be implemented in combination in a single embodiment. On the contrary, unless explicitly stated, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination.

[0121] Although the present disclosure has been described in the language of structural features and / or method actions, it is to be understood that the present disclosure defined in the appended claims is not necessarily limited to the above specific features or actions. Instead, the above specific features and actions are disclosed as example forms of implementing the claims.

[0122] In some aspects, a device for beam management is provided, comprising: at least one processor; and at least one memory, the at least one memory storing instructions that, when executed by the at least one processor, cause the device to: receive a continuous learning configuration from a network device, the continuous learning configuration indicating at least one of: a neighboring device, a first identifier of a first beam, or a second identifier of a second beam; transmit a side link message including data and the second identifier of the second beam to the neighboring device; and transmit the data to the network device via the first beam.

[0123] In some examples, at least one of the first beam or the second beam is used for exploration of the reinforcement learning model.

[0124] In some examples, the configuration also indicates a data type of the data transmitted to the network device based on a reinforcement learning model at the network device.

[0125] In some examples, the apparatus is caused to: receive a request for sidelink measurement from the network device; perform the sidelink measurement with at least one neighboring apparatus; and transmit results of the sidelink measurement regarding the at least one neighboring apparatus to the network device.

[0126] In some examples, the apparatus comprises a terminal device and the neighboring apparatus comprises another terminal device.

[0127] In some aspects, a device for beam management is provided, comprising: at least one processor; and at least one memory, the at least one memory storing instructions that, when executed by the at least one processor, cause the device to: receive a continuous learning configuration from a network device, the continuous learning configuration indicating at least one of: a first identifier of the device, a first beam, or a second identifier of a second beam; receive a side link message including data and the second identifier of the second beam from a neighboring device; and transmit the data to the network device via the second beam.

[0128] In some examples, at least one of the first beam or the second beam is used for exploration of the reinforcement learning model.

[0129] In some examples, the configuration also indicates a data type of the data transmitted to the network device based on a reinforcement learning model at the network device.

[0130] In some examples, the apparatus comprises a terminal device and the neighboring apparatus comprises another terminal device.

[0131] In some aspects, a device for beam management is provided, comprising: at least one processor; and at least one memory, the at least one memory storing instructions, which, when executed by the at least one processor, causes the device to: transmit a continuous learning configuration to a first terminal device, the continuous learning configuration indicating at least one of the following: a second terminal device, a first identifier of a first beam, or a second identifier of a second beam; transmit the continuous learning configuration to the second terminal device; receive data from the first terminal device via the first beam; and receive the data from the second terminal device via the second beam.

[0132] In some examples, the apparatus is caused to: transmit a request for sidelink measurement to the first terminal device; receive from the first terminal device a result of the sidelink measurement of at least one neighboring terminal device of the first terminal device; and determine the second terminal device from the at least one neighboring terminal device based on the result of the sidelink measurement received from the first terminal device.

[0133] In some examples, the apparatus is caused to: obtain information related to the data to be transmitted by the first terminal device; and obtain first terminal side link information related to the second terminal device.

[0134] In some examples, the device is caused to: select the first beam by a reinforcement learning model at the device; and select the second beam for exploration of the reinforcement learning model by the reinforcement learning model at the device.

[0135] In some examples, the apparatus is caused to update a reinforcement learning model at the apparatus based on the data received from the first terminal device and the data received from the second terminal device.

[0136] In some examples, the apparatus is a network device.

[0137] In some aspects, a method for beam management is provided, comprising: receiving, at a device, a continuous learning configuration from a network device, the continuous learning configuration indicating at least one of: a neighboring device, a first identifier of a first beam, or a second identifier of a second beam; transmitting a side link message including data and the second identifier of the second beam to the neighboring device; and transmitting the data to the network device via the first beam.

[0138] In some examples, at least one of the first beam or the second beam is used for exploration of the reinforcement learning model.

[0139] In some examples, the configuration further indicates a data type of the data transmitted to the network device based on a reinforcement learning model.

[0140] In some examples, the method further includes: receiving a request for sidelink measurement from the network device; performing the sidelink measurement with at least one neighboring device; and transmitting a result of the sidelink measurement about the at least one neighboring device to the network device.

[0141] In some examples, the apparatus comprises a terminal device and the neighboring apparatus comprises another terminal device.

[0142] In some aspects, a method for beam management is provided, comprising: receiving, at an apparatus, a continuous learning configuration from a network device, the continuous learning configuration indicating at least one of: the apparatus, a first identifier of a first beam, or a second identifier of a second beam; receiving a side link message including data and the second identifier of the second beam from a neighboring apparatus; and transmitting the data to the network device via the second beam.

[0143] In some examples, at least one of the first beam or the second beam is used for exploration of the reinforcement learning model.

[0144] In some examples, the configuration further indicates a data type of the data transmitted to the network device based on a reinforcement learning model.

[0145] In some examples, the apparatus comprises a terminal device and the neighboring apparatus comprises another terminal device.

[0146] In some aspects, a method for beam management is provided, comprising: transmitting, at an apparatus, a continuous learning configuration to a first terminal device, the continuous learning configuration indicating at least one of: a second terminal device, a first identifier of a first beam, or a second identifier of a second beam; transmitting the continuous learning configuration to a second terminal device; receiving data from the first terminal device via the first beam; and receiving the data from the second terminal device via the second beam.

[0147] In some examples, the method further includes: transmitting a request for sidelink measurement to the first terminal device; receiving a result of the sidelink measurement of at least one neighboring terminal device of the first terminal device from the first terminal device; and determining the second terminal device from the at least one neighboring terminal device based on the result of the sidelink measurement received from the first terminal device.

[0148] In some examples, it also includes: obtaining information related to the data to be transmitted by the first terminal device; and obtaining first terminal side link information related to the second terminal device.

[0149] In some examples, the method further includes: selecting the first beam through a reinforcement learning model at the device; and selecting the second beam for exploration of the reinforcement learning model through the reinforcement learning model at the device.

[0150] In some examples, it also includes: updating the reinforcement learning model at the apparatus based on the data received from the first terminal device and the data received from the second terminal device.

[0151] In some examples, the apparatus is a network device.

[0152] In some aspects, a computer-readable medium is provided, the computer-readable medium including instructions stored thereon, the instructions being used to cause an apparatus to at least perform a method according to some aspects of the present disclosure.

Claims

1. A device for beam management, comprising: at least one processor; as well as at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus to: receiving a continuous learning configuration from a network device, the continuous learning configuration indicating at least one of: a neighboring device, a first identifier of a first beam, or a second identifier of a second beam; transmitting a sidelink message including data and the second identifier of the second beam to the neighboring device; as well as The data is transmitted to the network device via the first beam.

2. The apparatus according to claim 1, wherein at least one of the first beam or the second beam is used for exploration of the reinforcement learning model. 3 . The apparatus of claim 1 , wherein the configuration further indicates a data type of the data transmitted to the network device based on a reinforcement learning model at the network device.

4. The apparatus of claim 1, wherein the apparatus is caused to: receiving a request for sidelink measurements from the network device; performing the sidelink measurements with at least one neighboring device; and The results of the sidelink measurements regarding the at least one neighboring device are transmitted to the network equipment. 5 . The apparatus of claim 1 , wherein the apparatus comprises a terminal device, and the neighboring apparatus comprises another terminal device.

6. A device comprising: at least one processor; as well as at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus to: receiving a continuous learning configuration from a network device, the continuous learning configuration indicating at least one of: the apparatus, a first identifier of a first beam, or a second identifier of a second beam; receiving a sidelink message including data and the second identification of the second beam from a neighboring device; as well as The data is transmitted to the network device via the second beam.

7. The apparatus according to claim 6, wherein at least one of the first beam or the second beam is used for exploration of the reinforcement learning model.

8. The apparatus of claim 6, wherein the configuration further indicates a data type of the data transmitted to the network device based on a reinforcement learning model at the network device.

9. The apparatus of claim 6, wherein the apparatus comprises a terminal device, and the neighboring apparatus comprises another terminal device.

10. An apparatus comprising: at least one processor; as well as at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus to: Transmitting a continuous learning configuration to the first terminal device, the continuous learning configuration indicating at least one of the following: a second terminal device, a first identifier of the first beam, or a second identifier of the second beam; transmitting the continuous learning configuration to the second terminal device; receiving data from the first terminal device via the first beam; as well as The data is received from the second terminal device via the second beam.

Images