Method for processing information and communication device

Abstract

The present disclosure provides a method for processing information and a communication device. The method comprises: determining an environmental state of an environment in which a network device is located at a current moment; inputting the environmental state at the current moment into a target deep learning model and outputting an action of the network device at the current moment; and executing the action at the current moment so as to carry out at least one of resource allocation and power control. The present disclosure provides a method, according to which a communication device can carry out resource allocation and power control on the basis of a DRL model, so that the resource allocation and the power control can be related to a dynamically changing environment. As a result, a policy can be adjusted in real time by means of interaction with the environment, so that the resource allocation and power control can better align with an environmental state, thereby achieving the maximization of the resource utilization rate.

Description

Information processing method and communication device TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of communication, and particularly relates to an information processing method and a communication device. BACKGROUND

[0002] In a scenario of fusion of non-terrestrial networks (NTN) and terrestrial cellular networks, the terrestrial cellular network can cause some interference to the ground station of the satellite, and how to reasonably utilize resources is a hot research issue.

[0003] SUMMARY

[0004] Embodiments of the present disclosure provide an information processing method and a communication device.

[0005] According to a first aspect of embodiments of the present disclosure, an information processing method is provided, applicable to a network device, and the method comprises:

[0006] determining an environment state of an environment in which the network device is located at a current time;

[0007] inputting the environment state at the current time into a target deep learning model to output an action of the network device at the current time;

[0008] performing the action at the current time to perform at least one of resource allocation and power control.

[0009] According to a second aspect of embodiments of the present disclosure, a communication device is provided, comprising:

[0010] a processing module configured to determine an environment state of an environment in which the network device is located at a current time;

[0011] inputting the environment state at the current time into a target deep learning model to output an action of the network device at the current time;

[0012] performing the action at the current time to perform at least one of resource allocation and power control.

[0013] According to a third aspect of embodiments of the present disclosure, a communication device is provided, comprising:

[0014] one or more processors;

[0015] The processor is configured to invoke instructions to cause the communication device to perform the information processing method of the first aspect.

[0016] According to a fourth aspect of embodiments of the present disclosure, a storage medium is provided, and the storage medium stores instructions, and the instructions are characterized in that, when the instructions run on a communication device, the communication device performs the information processing method of the first aspect.

[0017] According to a fifth aspect of the embodiments of the present disclosure, a program product is provided, which, when running on a communication device, causes the communication device to perform the information processing method according to the first aspect.

[0018] According to a sixth aspect of the embodiments of the present disclosure, a chip or chip system is provided. The chip or chip system comprises processing circuitry configured to perform the information processing method according to the first aspect described above.

[0019] In the above embodiments, the communication device can perform resource allocation and power control based on a deep reinforcement learning (DRL) model, so that the resource allocation and power control can be related to a dynamically changing environment, and thus can be adjusted in real time by interacting with the environment, so that the allocation of resources and power control can be more in line with the environment state, so as to maximize the utilization rate of resources.

[0020] It can be understood that the above communication device, storage medium, program product, chip or chip system are all used to perform the method proposed in the embodiments of the present disclosure. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0021] The above and / or additional aspects and advantages of the present disclosure will become apparent and more readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, in which:

[0022] FIGS. 1A-1B are one exemplary schematic diagram of an architecture of a communication system according to embodiments of the present disclosure.

[0023] FIG. 2A is one exemplary interaction schematic diagram of an information processing method according to embodiments of the present disclosure.

[0024] FIGS. 3A-3E are flow schematic diagrams of an information processing method according to embodiments of the present disclosure.

[0025] FIGS. 4A-4B are schematic diagrams of an information processing method according to embodiments of the present disclosure.

[0026] FIG. 5 is a structural schematic diagram of a communication device according to embodiments of the present disclosure.

[0027] FIG. 6A is a structural schematic diagram of a communication device according to one embodiment of the present disclosure.

[0028] FIG. 6B is a structural schematic diagram of a chip according to one embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] The embodiments of the present disclosure provide an information processing method and a communication device.

[0030] In a first aspect, the embodiments of the present disclosure provide an information processing method, which comprises:

[0031] determining an environment state of an environment at a current time at which the network device is located;

[0032] inputting the environment state at the current time into a target deep learning model to output an action of the network device at the current time;

[0033] performing the action at the current time to perform at least one of resource allocation and power control.

[0034] In the above embodiments, the network device can perform resource allocation and power control based on the DRL model, so that the resource allocation and power control can be related to the dynamically changing environment, and the strategy can be adjusted in real time through interaction with the environment, so that the resource allocation and power control can be more in line with the environment state, to maximize the resource utilization rate.

[0035] In combination with some embodiments of the first aspect, in some embodiments, the environment state at the current time is input into the target RB allocation model to output an RB allocation action of the communication device at the current time.

[0036] In combination with some embodiments of the first aspect, in some embodiments, performing the RB allocation action allocates resources to a ground communication device accessing the communication device.

[0037] In the above embodiments, the network device can perform resource allocation based on the target RB allocation model, so that the resource allocation strategy is related to the real-time environment, and the resource allocation can be more reasonable, to maximize the resource utilization rate.

[0038] In combination with some embodiments of the first aspect, in some embodiments, the ground communication device is communicated based on the resources allocated to the ground communication device.

[0039] In the above embodiments, the network device communicates with the ground communication device according to the allocated resources, and since the resource allocation considers the real-time environment, the interference can be reduced, and the accuracy of information transmission can be ensured.

[0040] In combination with some embodiments of the first aspect, in some embodiments, the environment state at the current time is input into a target power control model to output a power control action of the network device at the current time.

[0041] In combination with some embodiments of the first aspect, in some embodiments, the power control action is performed to adjust the power of the network device.

[0042] In the above embodiments, the network device can perform power control based on the target power control model, so that the real-time power of the network device can be highly related to the real-time environment, resource waste of the network device can be avoided, and energy consumption can be reduced.

[0043] In some embodiments of the first aspect, based on the interaction between the agent and the environment space, a data playback pool is constructed, the environment space including environment states of an environment in which the network device is located collected at multiple different time points; sample data is obtained from the data playback pool; and the deep learning model is trained based on the sample data to obtain the target DRL model.

[0044] In some embodiments of the first aspect, the sample data includes:

[0045] the environment state at the time t;

[0046] the action at the time t;

[0047] the environment state at the time t+1 after the action at the time t is performed;

[0048] the immediate reward at the time t+1 fed back after the action at the time t is performed.

[0049] In some embodiments of the first aspect, the environment state at the time t in the environment space is input into the first deep learning model, and the action at the time t is output; the action at the time t is fed back to the environment space to be performed to obtain the immediate reward at the time t+1; the environment state at the time t, the resource allocation action at the time t, and the immediate reward at the time t+1 and the environment state at the time t+1 are taken as sample data; and the data playback pool is obtained based on the sample data corresponding to multiple time points.

[0050] In the above embodiments, the sample data is collected through the interaction with the environment space, and samples are provided for subsequent model training.

[0051] In some embodiments of the first aspect, based on the interaction between the first RB allocation model in the agent and the environment space, a first data playback pool is constructed; first sample data is obtained from the first data playback pool; and the second RB allocation model is trained based on the first sample data using a deep Q network (DQN) algorithm to obtain a target RB allocation model.

[0052] In some embodiments of the first aspect, the first sample data includes:

[0053] the environment state at the time t;

[0054] the resource allocation action at the time t;

[0055] the environment state at the time t+1 after the resource allocation action at the time t is performed.

[0056] the instant reward of the t+1 moment fed back after the execution of the resource allocation action of the t moment.

[0057] In the above embodiment, through the interaction with the environment space, the collection of the sample data of the RB allocation model is realized, and samples are provided for the subsequent training of the RB allocation model.

[0058] In combination with some embodiments of the first aspect, in some embodiments, based on the interaction of the first power control model in the agent with the environment space, a second data playback pool is constructed; second sample data is obtained from the second data playback pool; the second power control model is trained based on the second sample data using a proximal policy optimization (PPO) algorithm to obtain a target power control model.

[0059] In combination with some embodiments of the first aspect, in some embodiments, the second sample data comprises:

[0060] the environment state of the t moment;

[0061] the power control action of the t moment;

[0062] the environment state of the t+1 moment after the execution of the power control action of the t moment;

[0063] the instant reward of the t+1 moment fed back after the execution of the power control action of the t moment.

[0064] In the above embodiment, through the interaction with the environment space, the collection of the sample data of the power control model is realized, and samples are provided for the subsequent training of the power control model.

[0065] In combination with some embodiments of the first aspect, in some embodiments, based on the agent and the environment space, the second RB allocation model and the second power control model are synchronously trained to obtain a target RB allocation model and a target power control model.

[0066] In the above embodiment, the RB allocation model and the power control model can be trained at the same time, the training efficiency can be improved, the same environment space is shared, and the utilization rate of resources is improved. In combination with some embodiments of the first aspect, in some embodiments, the environment state of the current moment is input into the target RB allocation model to output the RB allocation action of the current moment of the network device; the environment state of the current moment is input into the target power control model to output the power control action of the current moment of the network device.

[0067] In the above embodiment, the same environment state can be predicted by the target RB allocation model and the target power control model, so that the reasonable allocation of RB resources and the good power control of the network device can be realized.

[0068] In a second aspect, the embodiments of the present disclosure provide a network device, comprising:

[0069] a processing module configured to determine an environment state of an environment in which the network device is located at a current time, input the environment state at the current time into the target DRL model to output an action of the network device at the current time, and perform the action at the current time to perform at least one of resource allocation and power control.

[0070] According to a third aspect of the embodiments of the present disclosure, a communication device is provided, comprising:

[0071] one or more processors;

[0072] The processor is configured to invoke instructions to cause the communication device to perform the information processing method of the first aspect.

[0073] According to a fourth aspect of the embodiments of the present disclosure, a storage medium is provided, which stores instructions, and the instructions are configured to cause a communication device to perform the information processing method of the first aspect when the instructions are run on the communication device.

[0074] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, which is configured to cause a communication device to perform the communication method of the first aspect when the computer program product is run on the communication device.

[0075] According to a sixth aspect of the embodiments of the present disclosure, a chip or chip system is provided. The chip or chip system comprises a processing circuit configured to perform the method described in the optional implementation manner of the first aspect.

[0076] It can be understood that the above-mentioned communication device, storage medium, computer program product, chip or chip system are all used to perform the method provided by the embodiments of the present disclosure. Therefore, the beneficial effects that can be achieved are referred to the beneficial effects in the corresponding method, which will not be described here.

[0077] The embodiments of the present disclosure provide an information processing method. In some embodiments, the terms of determination method, information sending method, information receiving method, and communication method can be replaced with each other, the terms of information processing device, information sending device, information receiving device, and communication device can be replaced with each other, and the terms of information processing system, communication system, information sending system, and information receiving system can be replaced with each other.

[0078] The embodiments of the present disclosure are not exhaustive, but only illustrate some embodiments, and are not specific limitations on the protection scope of the present disclosure. In the case of no contradiction, each step in an embodiment can be implemented as an independent embodiment, and the steps can be combined arbitrarily, for example, the scheme after removing part of the steps in an embodiment can also be implemented as an independent embodiment, and the order of the steps in an embodiment can be exchanged arbitrarily, in addition, the optional implementation in an embodiment can be combined arbitrarily; in addition, the embodiments can be combined arbitrarily, for example, part or all steps of different embodiments can be combined arbitrarily, an embodiment can be combined with optional implementation of other embodiments.

[0079] In each embodiment of the present disclosure, the terms and / or descriptions between the embodiments are consistent if there is no special description and logical conflict, and can be referred to each other, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0080] The terms used in the embodiments of the present disclosure are only for the purpose of describing the specific embodiments, and not as a limitation on the present disclosure.

[0081] In the embodiments of the present disclosure, unless otherwise specified, the elements expressed in singular form, such as "one", "a", "the", "above", "said", "preceding", "this" and the like, can represent "one and only one", and can also represent "one or more", "at least one" and the like. For example, in the case of using articles such as "a", "an", "the" and the like in English, the noun after the article can be understood as singular expression, and can also be understood as plural expression.

[0082] In the embodiments of the present disclosure, "a plurality of" means two or more.

[0083] In some embodiments, the terms "at least one of", "at least one of", "at least one of", "one or more", "a plurality of", "multiple" and the like can be replaced with each other.

[0084] The description manner such as "at least one of A, B, C, …", "A and / or B and / or C, …" and the like in the embodiments of the present disclosure includes any one of A, B, C, … existing alone, and also includes any combination of any multiple of A, B, C, …, each of which can exist alone; for example, "at least one of A, B, C" includes a case of A alone, a case of B alone, a case of C alone, a case of combination of A and B, a case of combination of A and C, a case of combination of B and C, and a case of combination of A and B and C; for example, A and / or B includes a case of A alone, a case of B alone, and a case of combination of A and B.

[0085] In some embodiments, the description manner such as "A in a case, B in another case", "in response to a case A, in response to another case B" and the like can include the following technical solutions according to the case: A is executed regardless of B, that is, A in some embodiments; B is executed regardless of A, that is, B in some embodiments; A and B are selectively executed, that is, from A and B, execution is selected in some embodiments; A and B are both executed, that is, A and B in some embodiments. When there are more branches of A, B, C and the like, it is similar to the above.

[0086] The prefix words "first", "second" and the like in the embodiments of the present disclosure are only used to distinguish different description objects, and do not constitute limitation on the position, order, priority, quantity or content of the description objects. The description of the description objects should refer to the description in the context of the claims or embodiments, and should not constitute redundant limitation because of the use of the prefix words. For example, the description object is "field", and the ordinal words before "field" in "first field" and "second field" do not limit the position or order between "fields", and "first" and "second" do not limit whether the "fields" modified thereby are in the same message or not, nor limit the order of "first field" and "second field". For another example, the description object is "level", and the ordinal words before "level" in "first level" and "second level" do not limit the priority between "levels". For another example, the quantity of the description object is not limited by the ordinal words, and can be one or more. For example, "first device", wherein the quantity of "device" can be one or more. In addition, the objects modified by different prefix words can be the same or different, for example, the description object is "device", and "first device" and "second device" can be the same device or different devices, and the types thereof can be the same or different; for another example, the description object is "information", and "first information" and "second information" can be the same information or different information, and the contents thereof can be the same or different.

[0087] In some embodiments, "including A", "containing A", "for indicating A", "carrying A" can be interpreted as directly carrying A, or indirectly indicating A.

[0088] In some embodiments, the terms “in response to,” “in response to determining,” “in the event that,” “when,” “if,” “upon,” and the like can be replaced with each other.

[0089] In some embodiments, the terms “greater than,” “greater than or equal to,” “not less than,” “more than,” “more than or equal to,” “not less than,” “higher than,” “higher than or equal to,” “not lower than,” “above,” and the like can be replaced with each other, and the terms “less than,” “less than or equal to,” “not greater than,” “less than,” “less than or equal to,” “not more than,” “lower than,” “lower than or equal to,” “not higher than,” “below,” and the like can be replaced with each other.

[0090] In some embodiments, an apparatus and the like can be interpreted as an entity, and can also be interpreted as virtual, and the name thereof is not limited to the name described in the embodiments, and the terms “apparatus,” “equipment,” “device,” “circuit,” “network element,” “node,” “function,” “unit,” “section,” “system,” “network,” “chip,” “chip system,” “entity,” “subject,” and the like can be replaced with each other.

[0091] In some embodiments, “network” can be interpreted as an apparatus (for example, an access network device, a core network device, and the like) included in the network.

[0092] In some embodiments, the terms “access network device (AN device),” “radio access network device (RAN device),” “base station (BS),” “radio base station,” “fixed station,” “node,” “access point,” “transmission point (TP),” “reception point (RP),” “transmission / reception point (TRP),” “panel,” “antenna panel,” “antenna array,” “cell,” “macro cell,” “small cell,” “femto cell,” “pico cell,” “sector,” “cell group,” “carrier,” “component carrier,” “bandwidth part (BWP),” and the like can be used interchangeably.

[0093] In some embodiments, the terms "terminal," "terminal device," "user equipment (UE)," "user terminal," "mobile station (MS)," "mobile terminal (MT)," "subscriber station," "mobile unit," "subscriber unit," "wireless unit," "remote unit," "mobile device," "wireless device," "wireless communication device," "remote device," "mobile subscriber station," "access terminal," "mobile terminal," "wireless terminal," "remote terminal," "handset," "user agent," "mobile client," "client," and so on can be replaced with each other.

[0094] In some embodiments, an access network device, a core network device, or a network device can be replaced with a terminal. For example, for a structure in which communication between an access network device, a core network device, or a network device and a terminal is replaced with communication between a plurality of terminals (for example, also referred to as device-to-device (D2D), vehicle-to-everything (V2X), and so on), embodiments of the present disclosure can also be applied. In this case, a structure in which a terminal has all or part of the functions of an access network device can also be provided. Furthermore, the language of "uplink," "downlink," and so on can also be replaced with language corresponding to communication between terminals (for example, "side"). For example, an uplink channel, a downlink channel, and so on can be replaced with a side channel, and an uplink, a downlink, and so on can be replaced with a side link.

[0095] In some embodiments, a terminal can be replaced with an access network device, a core network device, or a network device. In this case, a structure in which an access network device, a core network device, or a network device has all or part of the functions of a terminal can also be provided.

[0096] In some embodiments, the data, information, etc. can be obtained in compliance with the laws and regulations of the country where the location is situated.

[0097] In some embodiments, the data, information, etc. can be obtained after obtaining the consent of the user.

[0098] In addition, each element, each row, or each column in the table of the embodiments of the present disclosure can be implemented as an independent embodiment, and any combination of any element, any row, or any column can also be implemented as an independent embodiment.

[0099] The correspondence shown in each table in the present disclosure can be configured or predefined. The values of the information in each table are merely examples, and other values can be configured, and the present disclosure is not limited. When configuring the correspondence between the information and each parameter, it is not necessarily required to configure all the correspondences shown in each table. For example, the correspondences shown in some rows in the table in the present disclosure can also not be configured. For another example, the above table can be appropriately deformed, adjusted, etc., such as splitting, merging, etc. The names of the parameters shown in the titles of the above tables can also use other names understandable by the communication device, and the values or representations of the parameters can also use other values or representations understandable by the communication device. The above tables can also use other data structures when implemented, such as arrays, queues, containers, stacks, linear tables, pointers, linked lists, trees, graphs, structures, classes, heaps, hash tables, etc.

[0100] The predefinition in the present disclosure can be understood as defining, predefining, storing, pre-storing, pre-negotiating, pre-configuring, solidifying, or pre-burning.

[0101] FIG. 1A is a schematic diagram of an architecture of a communication system according to an embodiment of the present disclosure. As shown in FIG. 1A, the communication system 100 can include a terminal 101, an access network device 102, a core network device 103, a satellite 104, and a ground station 105. Among them, the access network device 102 and the core network device 103 constitute a ground cellular network, and the satellite 104, the ground station 105, and the core network device 103 can constitute an NTN.

[0102] In some embodiments, the terminal device 101 includes at least one of a mobile phone, a wearable device, an Internet of Things device, a communication-capable automobile, a smart automobile, a tablet (Pad), a wireless transceiver-equipped computer, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal device in industrial control, a wireless terminal device in self-driving, a wireless terminal device in remote medical surgery, a wireless terminal device in a smart grid, a wireless terminal device in transportation safety, a wireless terminal device in a smart city, a wireless terminal device in a smart home, and the like, but is not limited thereto.

[0103] In some embodiments, the access network device 102 is at least one of a node or device that accesses a terminal to a wireless network, and can include an evolved NodeB (eNB) in a 5G communication system, a next generation eNB (ng-eNB), a next generation NodeB (gNB), a node B (NB), a home node B (HNB), a home evolved node B (HeNB), a wireless backhaul device, a radio network controller (RNC), a base station controller (BSC), a base transceiver station (BTS), a base band unit (BBU), a mobile switching center, a base station in a 6G communication system, an Open RAN, a Cloud RAN, a base station in other communication systems, an access node in a Wi-Fi system, and the like, but is not limited thereto.

[0104] In some embodiments, the technical solutions of the present disclosure can be applied to an Open RAN architecture, in which case, the interfaces between or within the access network devices involved in the embodiments of the present disclosure can become internal interfaces of the Open RAN, and the processes and information interactions between these internal interfaces can be implemented through software or programs.

[0105] In some embodiments, the access network device can be composed of a central unit (CU) and a distributed unit (DU), where the CU can also be referred to as a control unit. The CU-DU structure can split the protocol layers of the access network device, with some of the protocol layers being controlled by the CU and the rest of the protocol layers or all of the protocol layers being distributed in the DUs and controlled by the CU, but is not limited thereto.

[0106] In some embodiments, the core network device 103 can be one device including the first network element, the second network element, etc., or can be multiple devices or device groups including all or part of the first network element, the second network element, etc. respectively. The network element can be virtual or physical. The core network includes at least one of an evolved packet core (EPC), a 5G core network (5GCN), and a next generation core (NGC), for example.

[0107] Optionally, the core network device can include one or more of a location management function (LMF) network element, a user plane function (UPF) network element, a policy control function (PCF) network element, an application function (AF) network element, a multicast / broadcast user plane function (MB-UPF) network element, a multicast / broadcast session management function (MB-SMF) network element, a multicast / broadcast service function (MB-SF) network element, a multicast / broadcast service transport function (MBSTF) network element, an access and mobility management function (AMF) network element, a session management function (SMF) network element, and a unified data management (UDM) network element.

[0108] In some embodiments, the satellite 104 includes, for example, low earth orbit satellites, sun-synchronous orbit satellites, large elliptical orbit satellites, geosynchronous orbit satellites, geostationary orbit satellites, communication satellites, navigation satellites, remote sensing satellites, and scientific satellites, etc.

[0109] In some embodiments, the ground station 105 includes, for an any satellite communication line, a transmitting and receiving ground station, an uplink and downlink, and a communication satellite transponder. The basic function of the ground station 105 is to transmit signals to the satellite, and to receive signals transmitted by other ground stations through the satellite. The ground station 105 can include, for example, fixed stations, mobile stations, and detachable stations, etc.

[0110] It can be understood that the communication system described in the embodiments of the present disclosure is for more clearly illustrating the technical solutions of the embodiments of the present disclosure, and does not constitute a limitation on the technical solutions proposed by the embodiments of the present disclosure. Those skilled in the art can know that, as the system architecture evolves and new business scenarios appear, the technical solutions proposed by the embodiments of the present disclosure are also applicable to similar technical problems.

[0111] The following embodiments of the present disclosure can be applied to the communication system 100 shown in FIG. 1A or part of the subject, but are not limited thereto. The subjects shown in FIG. 1A are exemplary, and the communication system can include all or part of the subjects in FIG. 1A, or other subjects other than those in FIG. 1A. The number and form of each subject is arbitrary, and the connection relationship between the subjects is exemplary. The subjects can be connected or not connected, and the connection can be in any way, can be direct connection or indirect connection, can be wired connection or wireless connection.

[0112] Embodiments of the present disclosure can be applied to Long Term Evolution (LTE), LTE-Advanced (LTE-A), LTE-Beyond (LTE-B), SUPER 3G, IMT-Advanced, 4th generation mobile communication system (4G), 5th generation mobile communication system (5G), 5G new radio (NR), Future Radio Access (FRA), New-Radio Access Technology (RAT), New Radio (NR), New radio access (NX), Future generation radio access (FX), Global System for Mobile communications (GSM (registered trademark)), CDMA2000, Ultra Mobile Broadband (UMB), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.20, Ultra-WideBand (UWB), Bluetooth (Bluetooth (registered trademark)), Public Land Mobile Network (PLMN) network, Device-to-Device (D2D) system, Machine to Machine (M2M) system, Internet of Things (IoT) system, Vehicle-to-Everything (V2X), system using other determination methods, next-generation system expanded based thereon, and the like. Further, a plurality of systems can be applied in combination (for example, combination of LTE or LTE-A and 5G, and the like).

[0113] Figure 1B is a schematic diagram of a communication system shown in an embodiment of the present disclosure. As shown in Figure 1B, the UE can have two communication modes: NTN network and terrestrial cellular network. Among them, the low-orbit satellite can divide K ground stations into a group (Group) to provide services for them. K ground stations can share a resource block (RB), such as RB1, RB2, etc. There are K ground stations in a group of ground stations, and at the same time, K messages belonging to each ground station can be sent, and some common messages of K ground stations can also be sent. As shown in Figure 1B, the resources in RB1 and RB2 can be divided into common part resources and private part resources, among which the common part resources can be used to transmit some common messages of K ground stations, and the private part resources can be used to transmit the respective messages of K ground stations.

[0114] In actual applications, there are also situations where terrestrial cellular networks and satellites share the same frequency. As shown in Figure 1B, T-cell 2 and Group 1 use the same frequency, and T-cell 1 and Group 2 use the same frequency. To improve resource utilization, it is necessary to properly allocate resources and control power of network equipment.

[0115] FIG2A is an interactive diagram of an information processing method according to an embodiment of the present disclosure. As shown in FIG2A , the present disclosure embodiment relates to an information processing method, which includes:

[0116] Step S2101: construct a first data replay pool and a second data replay pool based on the interaction between the agent and the environment space.

[0117] In some embodiments, the environmental space includes environmental states of the environment in which the network device is located collected at different times.

[0118] In some embodiments, the environmental status of the network device may include, but is not limited to: the connection status of the network device, signal quality, data transmission speed, etc.

[0119] In some embodiments, the network device may be a satellite, a ground station, or a gateway in a non-terrestrial network (NTN).

[0120] In some embodiments, the network device may be an access network device or a core network device in a terrestrial cellular network.

[0121] In some embodiments, an agent includes a first RB allocation model and a first power control model, and the agent interacts with the environment space to construct a first data playback pool and a second data playback pool.

[0122] In some embodiments, the first data replay pool comprises first sample data for training the second RB allocation model.

[0123] In some embodiments, the second data replay pool comprises second sample data for training the second power control model.

[0124] In some embodiments, the RB allocation model and the power control model are Deep reinforcement learning (DRL) models.

[0125] In some embodiments, based on the interaction of the agent with the environment space, the process of constructing the data replay pool can include: inputting the environment state at time t in the environment space into the DRL model in the agent, outputting the action at time t, feeding back the action at time t to the environment space to execute to obtain the immediate reward at time t+1; taking the environment state at time t, the action at time t, and the immediate reward at time t+1 and the environment state at time t+1 as sample data, and obtaining the data replay pool based on the sample data corresponding to multiple time points.

[0126] It can be understood that action: each agent takes its own action according to the monitored local environment state to determine the allocation of the corresponding RB and the subsequent power control.

[0127] atateAt time t, the agent determines the action a according to the observed state staet t Take action, then the environment will transition to the next state staet t+1 , and feedback a reward reward t+1 . Wherein each state contains useful information, which can enable the agent to make appropriate decisions, i.e. action.

[0128] reward:agent in state staet t Obtain immediate reward reward t+1 after performing the action, which is an incentive for the agent to train the optimal strategy, which can select appropriate actions to achieve the pre-specified goal.

[0129] In some embodiments, based on the interaction between the first RB allocation model in the agent and the environment space, a first data replay pool is constructed, that is, the environment state at time t in the environment space is input into the first RB allocation model in the agent, the action at time t is output, the resource allocation action at time t is fed back to the environment space to execute to obtain the immediate reward at time t+1; the environment state at time t, the resource allocation action at time t, and the immediate reward at time t+1 and the environment state at time t+1 are taken as sample data, and the first data replay pool is obtained based on the sample data corresponding to multiple times.

[0130] In some embodiments, based on the interaction between the first power control model in the agent and the environment space, a second data replay pool is constructed, that is, the environment state at time t in the environment space is input into the first RB allocation model in the agent, the action at time t is output, the power control action at time t is fed back to the environment space to execute to obtain the immediate reward at time t+1; the environment state at time t, the power control action at time t, and the immediate reward at time t+1 and the environment state at time t+1 are taken as sample data, and the second data replay pool is obtained based on the sample data corresponding to multiple times.

[0131] Step S2102, first sample data is obtained from the first data replay pool, and the second RB allocation model is trained based on the first sample data to obtain a target RB allocation model.

[0132] In some embodiments, the first data replay pool includes multiple first sample data. In each training, a first sample data can be determined from the first data replay pool, for example, a first sample data can be sampled from the first data replay pool, and further, the second RB allocation model is trained based on the first sample data to obtain a target RB allocation model.

[0133] In some embodiments, each first sample data includes:

[0134] the environment state at time t;

[0135] the resource allocation action at time t;

[0136] the environment state at time t+1 after the resource allocation action at time t is executed;

[0137] the immediate reward at time t+1 fed back after the resource allocation action at time t is executed.

[0138] In some embodiments, since the action of RB allocation is discrete, the second RB allocation model can be trained based on the first sample data according to a deep Q-network (DQN) algorithm until a training end condition is met to obtain a target RB allocation model.

[0139] In some embodiments, the algorithm for training the second RB allocation model includes, but is not limited to, a Policy Gradient algorithm, a Double DQN algorithm, an Actor-Critic algorithm, a State-Action-Reward-State-Action (SARSA) algorithm, and a Trust Region Policy Optimization (TRPO) algorithm.

[0140] In some embodiments, the training end condition of the second RB allocation model includes, but is not limited to, that the RB allocation model reaches a preset number of training rounds, that the RB allocation model accuracy and loss reaches a preset threshold, that the RB allocation model converges (for example, the loss function value of the model tends to be stable and no longer decreases), and the like.

[0141] In step S2103, a second sample data is obtained from the second data replay pool, and a second power control model is trained based on the second sample data to obtain a target power control model.

[0142] In some embodiments, the second data replay pool includes a plurality of second sample data. A second sample data can be determined from the second data replay pool at each training, for example, a second sample data can be sampled from the second data replay pool at random, and further, the second power control model is trained based on the second sample data to obtain the target power control model.

[0143] In some embodiments, the second sample data includes:

[0144] an environment state at time t;

[0145] a power control action at time t;

[0146] an environment state at time t+1 after the power control action at time t is performed;

[0147] an immediate reward at time t+1 fed back after the power control action at time t is performed.

[0148] In some embodiments, since the power control action is continuous, the second power control model can be trained based on the second sample data according to a proximal policy optimization (PPO) algorithm until a training end condition is met to obtain the target power control model.

[0149] In some embodiments, the algorithm for training the second power control model includes but is not limited to: a policy gradient algorithm, a double DQN algorithm, a policy network (actor)-value network (critic) algorithm, an algorithm for learning a state-action-reward-state-action (SARSA) Markov decision process policy, and a trust region policy optimization (TRPO) algorithm.

[0150] In some embodiments, the training end condition of the second power control model can include but is not limited to: the power control model reaches a preset training round, the power control allocation model accuracy and loss reaches a preset threshold, the power control allocation model converges (for example, the loss function value of the model tends to be stable and no longer decreases), and the like.

[0151] Step S2104, determining the environment state of the network device at the current time.

[0152] In some embodiments, the network device can collect the environment state at the current time.

[0153] In some embodiments, the satellite can collect the environment state at the current time. Optionally, the environment state at the current time can include but is not limited to: the connection condition of the satellite, the signal quality, the data transmission speed, and the like.

[0154] Step S2105, inputting the environment state at the current time into the target RB allocation model, and outputting the RB allocation action of the network device at the current time.

[0155] In some embodiments, the environment state of the current environment can be input into the trained target RB allocation model, and the target RB allocation model can make resource allocation decisions based on the environment state at the current time to obtain the RB allocation action of the network device at the current time.

[0156] In some embodiments, the RB allocation action can include but is not limited to at least one of the following strategies:

[0157] The time-frequency resource occupied by the network device;

[0158] The frequency domain resource occupied by the network device;

[0159] The allocation strategy of the time-frequency resource of the ground communication device covered by the network device.

[0160] In some embodiments, the network device occupies at least one RB, and the target RB allocation model can output the resource allocation action of the RB.

[0161] Optionally, the resource allocation action of the RB can include: a first resource set and a second resource set in each RB, wherein the first resource set is a shared resource part, and the second resource set is a private resource part.

[0162] Optionally, the resource allocation action of the RB can include: a corresponding device group of each RB, and a mapping relationship between K ground stations or UEs in the device group and resources in the second resource set of the corresponding RB, that is, resources in the second resource set of the corresponding RB can be allocated to the K ground stations or UEs.

[0163] In step S2106, a ground communication device of the access network device is allocated resources by performing the RB allocation action.

[0164] In some embodiments, after obtaining the RB allocation action, resources occupied by each ground communication device can be determined based on the RB allocation action, and the ground communication device can be indicated.

[0165] In some embodiments, the ground communication device can be indicated as being allocated resources by the resource configuration information.

[0166] In some embodiments, when the network device is a satellite, the ground communication device can include but is not limited to a ground station and a UE.

[0167] In step S2107, the network device communicates with the ground communication device.

[0168] In some embodiments, the communication transmission can include but is not limited to transmission of information, data or messages.

[0169] In some embodiments, the network device can perform uplink transmission based on the resources allocated to the ground communication device, for example, which can include but is not limited to transmission of uplink information, uplink data or uplink messages.

[0170] In some embodiments, the network device can perform downlink transmission based on the resources allocated to the ground communication device, for example, which can include but is not limited to transmission of downlink information, downlink data or downlink messages.

[0171] In some embodiments, the network device can perform common message transmission with multiple ground communication information based on the shared resources.

[0172] In some embodiments, the network device can perform separate message transmission based on the private resources of the ground communication device.

[0173] In step S2108, the environment state at the current time is input into the target power control model, and a power control action of the network device is output.

[0174] In some embodiments, the environment state of the current environment can be input into the trained target power control model, and the target power control model can make a power control decision based on the environment state at the current time to obtain a power control action of the network device at the current time.

[0175] In some embodiments, the power control action can include, but is not limited to, at least one of the following strategies:

[0176] a power adjustment coefficient or step size of the network device;

[0177] an adjustment direction of the network device, for example, upward adjustment, or downward adjustment;

[0178] a target power of the network device after adjustment;

[0179] a power adjustment duration of the network device.

[0180] Step S2109: performing the power control action to adjust the power of the network device.

[0181] In some embodiments, the current power of the network device can be adjusted upward.

[0182] In some embodiments, the current power of the network device can be adjusted downward according to the power adjustment coefficient or step size.

[0183] In some embodiments, the power of the network device can be monitored until the adjusted power of the network device is the target power.

[0184] In some embodiments, the power of the network device can be adjusted to reach the target power within the power adjustment duration.

[0185] In some embodiments, the current power of the network device can be adjusted according to the power adjustment coefficient or step size and the adjustment direction within the power adjustment duration.

[0186] In some embodiments, the names of information and the like are not limited to the names described in the embodiments, and terms such as "information", "message", "signal", "signaling", "report", "configuration", "indication", "instruction", "command", "channel", "parameter", "field", "symbol", "symbol", "codebook", "codeword", "codepoint", "bit", "data", "program", "chip", and the like can be replaced with each other.

[0187] In some embodiments, terms such as "uplink", "uplink", "physical uplink", and the like can be replaced with each other, terms such as "downlink", "downlink", "physical downlink" and the like can be replaced with each other, terms such as "side", "sidelink", "sidelink communication", "sidelink communication", "direct connection", "direct connection link", "direct connection", "direct connection link communication" and the like can be replaced with each other.

[0188] In some embodiments, terms such as "downlink control information (DCI)", "downlink (DL) assignment", "DL DCI", "uplink (UL) grant", "UL DCI", and the like can be replaced with each other.

[0189] In some embodiments, terms such as "physical downlink shared channel (PDSCH)", "DL data", and the like can be replaced with each other, and terms such as "physical uplink shared channel (PUSCH)", "UL data", and the like can be replaced with each other.

[0190] In some embodiments, terms such as "radio", "wireless", "radio access network (RAN)", "access network (AN)", "RAN-based" and the like can be replaced with each other.

[0191] In some embodiments, the terms of wireless access scheme, waveform, and the like can be replaced with each other.

[0192] In some embodiments, the terms of “acquire”, “obtain”, “get”, “receive”, “transmit”, “bidirectional transmission”, “send and / or receive”, and the like can be replaced with each other, which can be interpreted as receiving from other subjects, acquiring from protocols, acquiring from higher layers, obtaining by self-processing, implementing autonomously, and the like.

[0193] In some embodiments, the terms of “send”, “transmit”, “report”, “issue”, “transmit”, “bidirectional transmission”, “send and / or receive”, and the like can be replaced with each other.

[0194] In some embodiments, the terms of “certain”, “preseted”, “preset”, “set”, “indicated”, “certain”, “arbitrary”, “first”, and the like can be replaced with each other, “certain A”, “preset A”, “preset A”, “set A”, “indicated A”, “certain A”, “arbitrary A”, “first A” can be interpreted as A specified in advance in protocols and the like, can be interpreted as A obtained by setting, configuring, or indicating, and the like, can be interpreted as certain A, certain A, arbitrary A, or first A, and the like, but are not limited thereto.

[0195] In some embodiments, the determination or judgment can be performed by a value represented by 1 bit (0 or 1), can be performed by a true or false value (Boolean value) represented by true or false, can be performed by comparison of numerical values (for example, comparison with a predetermined value), but is not limited thereto.

[0196] In some embodiments, “not expecting to receive” can be interpreted as not receiving on time domain resources and / or frequency domain resources, can be interpreted as not performing subsequent processing on the data and the like after receiving the data and the like; “not expecting to send” can be interpreted as not sending, can be interpreted as sending but not expecting the receiving party to respond to the content of the sending.

[0197] The information processing method related by the embodiments of the present disclosure can include at least one of steps S2101-S2109. For example, step S2101 can be implemented as an independent embodiment, step S2102 can be implemented as an independent embodiment, step S2103 can be implemented as an independent embodiment, steps S2101+S2102+S2103 can be implemented as an independent embodiment, steps S2104+S2105 can be implemented as an independent embodiment, steps S2104+S2105+S2106 can be implemented as an independent embodiment, steps S2104+S2105+S2106+S2017 can be implemented as an independent embodiment, steps S2104+S2108 can be implemented as an independent embodiment, steps S2104+S2108+S2109 can be implemented as an independent embodiment, but not limited thereto.

[0198] In some embodiments, the order of steps S2105, S2106, S2107, and S2108, S2109 can be exchanged.

[0199] In some embodiments, steps S2105, S2106, S2107, and S2108, S2109 are optional, and one or more of these steps can be omitted or replaced in different embodiments.

[0200] In some embodiments, steps S2101, S2102, and S2013 are optional, and one or more of these steps can be omitted or replaced in different embodiments.

[0201] FIG. 3A is a processing schematic diagram of an information processing method according to an embodiment of the present disclosure. As shown in FIG. 3A, the embodiments of the present disclosure relate to an information processing method, and the method includes:

[0202] Step S3101, based on the interaction of the agent with the environment space, constructing a data playback pool.

[0203] In some embodiments, the data playback pool can store a plurality of sample data, and optionally, the sample data can include:

[0204] the environment state at time t;

[0205] the action at time t;

[0206] the environment state at time t+1 after performing the action at time t;

[0207] the immediate reward at time t+1 after performing the action at time t.

[0208] In some embodiments, the environment state at time t in the environment space is input into the DRL model in the agent, an action at time t is output, the action at time t is fed back to the environment space to be executed to obtain an immediate reward at time t+1, the environment state at time t, the action at time t, the immediate reward at time t+1, and the environment state at time t+1 are taken as sample data, a data replay pool is obtained based on the sample data corresponding to multiple times, that is, the sample data corresponding to multiple times is stored in the data replay pool.

[0209] In step S3102, sample data is obtained from the data replay pool, and the DRL model is trained based on the sample data to obtain a target DRL model.

[0210] In some embodiments, the data replay pool can be a first data replay pool, the DRL model can be an RB allocation model, first sample data can be obtained from the first data replay pool, and the RB allocation model can be trained based on the first sample data of the first data replay pool to obtain a target RB allocation model.

[0211] In some embodiments, the data replay pool can be a second data replay pool, the DRL model can be a power control model, second sample data can be obtained from the second data replay pool, and the power control model can be trained based on the second sample data of the second data replay pool to obtain a target power control model.

[0212] In some embodiments, the data replay pool can include a first data replay pool and a second data replay pool. Optionally, first sample data can be obtained from the first data replay pool, and the RB allocation model can be trained based on the first sample data of the first data replay pool to obtain a target RB allocation model, and second sample data can be obtained from the second data replay pool, and the power control model can be trained based on the second sample data of the second data replay pool to obtain a target power control model. That is, the RB allocation model and the power control model are jointly trained based on the agent and the environment space.

[0213] The optional implementation of step S3102 can refer to the optional implementation of steps S2102-S2103 of FIG. 2A and other associated parts in the embodiments involved in FIG. 2A, which will not be described here.

[0214] In step S3103, the environment state at the current time of the network device is determined.

[0215] The optional implementation of step S3103 can refer to the optional implementation of step S2104 of FIG. 2A and other associated parts in the embodiments involved in FIG. 2A, which will not be described here.

[0216] Step S3104, inputting the environment state at the current moment into the target DRL model to output a network device action at the current moment.

[0217] In some embodiments, the target DRL model is a target RB allocation model, inputting the environment state at the current moment into the target RB allocation model can output an RB allocation action of the network device at the current moment by the target RB allocation model, which can be referred to the optional implementation of step S2105 in FIG. 2A and other associated parts in the embodiments involved in FIG. 2A, and details are not described herein.

[0218] In some embodiments, the target DRL model is a target power control model, inputting the environment state at the current moment into the target power control model can output a power control action of the network device at the current moment by the target power control model, which can be referred to the optional implementation of step S2108 in FIG. 2A and other associated parts in the embodiments involved in FIG. 2A, and details are not described herein.

[0219] Step S3105, performing the action at the current moment to perform at least one of resource allocation and power control.

[0220] In some embodiments, after obtaining the RB allocation action, the resource occupied by each ground communication device can be determined based on the RB allocation action, and the ground communication device can be indicated, which can be referred to the optional implementation of steps S2106-S2107 in FIG. 2A and other associated parts in the embodiments involved in FIG. 2A, and details are not described herein.

[0221] In some embodiments, after obtaining the power control action, the network device can be adjusted in power based on the power control action, which can be referred to the optional implementation of step S2109 in FIG. 2A and other associated parts in the embodiments involved in FIG. 2A, and details are not described herein.

[0222] The information processing method involved in the embodiments of the present disclosure can include at least one of steps S3101-S2105. For example, step S3101 can be implemented as an independent embodiment, step S3102 can be implemented as an independent embodiment, steps S3103+S3104 can be implemented as an independent embodiment, steps S3103+S3104+S3105 can be implemented as an independent embodiment, but not limited thereto.

[0223] In the present embodiment or the present embodiment, each step can be independent, arbitrarily combined or exchanged in order, the optional mode or the optional example can be arbitrarily combined, and can be arbitrarily combined with any step of other embodiments or other embodiments.

[0224] FIG. 3B is a flowchart of an information processing method according to an embodiment of the present disclosure. As shown in FIG. 2B, the embodiment of the present disclosure relates to an information processing method, and the method comprises:

[0225] In step S3201, a first data playback pool is constructed based on the interaction between the first RB allocation model in the agent and the environment space.

[0226] The optional implementation of step S3201 can refer to the optional implementation of step S2102 in FIG. 2A and other associated parts in the embodiments related to FIG. 2A, which will not be repeated here.

[0227] In step S3202, first sample data is obtained from the first data playback pool, and the second RB allocation model is trained based on the first sample data to obtain a target RB allocation model.

[0228] The optional implementation of step S3202 can refer to the optional implementation of step S2102 in FIG. 2A and other associated parts in the embodiments related to FIG. 2A, which will not be repeated here.

[0229] In step S3203, the environment state of the network device at the current time is determined.

[0230] The optional implementation of step S2203 can refer to the optional implementation of step S2104 in FIG. 2A and other associated parts in the embodiments related to FIG. 2A, which will not be repeated here.

[0231] In step S3204, the environment state at the current time is input into the target RB allocation model to output the RB allocation action of the network device at the current time.

[0232] The optional implementation of step S3204 can refer to the optional implementation of step S2105 in FIG. 2A and other associated parts in the embodiments related to FIG. 2A, which will not be repeated here.

[0233] In step S3205, the RB allocation action is performed to allocate resources to the ground communication device accessing the network device.

[0234] The optional implementation of step S3205 can refer to the optional implementation of step S2106 in FIG. 2A and other associated parts in the embodiments related to FIG. 2A, which will not be repeated here.

[0235] In step S3206, the resource allocated to the ground communication device is used for communication transmission with the ground communication device.

[0236] The optional implementation of step S3206 can refer to the optional implementation of step S2107 in FIG. 2A and other associated parts in the embodiments related to FIG. 2A, which will not be repeated here.

[0237] The information processing method related to the embodiments of the present disclosure can include at least one of steps S3201-S3206. For example, step S3201 can be implemented as an independent embodiment, step S3202 can be implemented as an independent embodiment, steps S3203+S2204 can be implemented as an independent embodiment, steps S3203+S3204+S3205 can be implemented as an independent embodiment, steps S3203+S3204+S3205+S3206 can be implemented as an independent embodiment, but are not limited thereto.

[0238] In the present embodiment or example, each step can be independently combined or exchanged in order, and optional modes or examples can be combined with any step of other embodiments or other examples, without contradiction.

[0239] FIG. 3C is a flow diagram of an information processing method according to an embodiment of the present disclosure. As shown in FIG. 3C, the embodiments of the present disclosure relate to an information processing method, and the method comprises:

[0240] Step S3301, based on the interaction between the first power control model in the agent and the environment space, a second data playback pool is constructed.

[0241] The optional implementation of step S3301 can refer to the optional implementation of step S2102 of FIG. 2A and other related parts in the embodiments related to FIG. 2A, which will not be repeated here.

[0242] Step S3302, obtaining second sample data from the second data playback pool, and training the second power control model based on the second sample data to obtain a target power control model.

[0243] The optional implementation of step S3302 can refer to the optional implementation of step S2103 of FIG. 2A and other related parts in the embodiments related to FIG. 2A, which will not be repeated here.

[0244] Step S3303, determining the current environmental state of the environment where the network device is located.

[0245] The optional implementation of step S3303 can refer to the optional implementation of step S2104 of FIG. 2A and other related parts in the embodiments related to FIG. 2A, which will not be repeated here.

[0246] Step S3304, inputting the current environmental state into the target power control model, and outputting the power control action of the network device.

[0247] The optional implementation of step S3304 can refer to the optional implementation of step S2108 in FIG. 2A and other associated parts in the embodiments involved in FIG. 2A, which will not be repeated here.

[0248] Step S3305: performing a power control action to adjust the power of the network device.

[0249] The optional implementation of step S3305 can refer to the optional implementation of step S2109 in FIG. 2A and other associated parts in the embodiments involved in FIG. 2A, which will not be repeated here.

[0250] The information processing method involved in the embodiments of the present disclosure can include at least one of steps S3301-S3304. For example, step S3301 can be implemented as an independent embodiment, step S3302 can be implemented as an independent embodiment, steps S3303+S3305 can be implemented as an independent embodiment, and steps S3303+S3304+S3305 can be implemented as an independent embodiment, but are not limited thereto.

[0251] In the present embodiment or example, each step can be independently combined or exchanged in order without contradiction, and the optional mode or optional example can be combined with any step of other embodiments or other examples.

[0252] FIG. 3D is a flow diagram of an information processing method according to an embodiment of the present disclosure. As shown in FIG. 3D, the embodiments of the present disclosure involve an information processing method, and the method includes:

[0253] Step S3401: determining the environment state of the network device at the current time.

[0254] The optional implementation of step S3401 can refer to the optional implementation of step S2104 in FIG. 2A and other associated parts in the embodiments involved in FIG. 2A, which will not be repeated here.

[0255] Step S3402: inputting the environment state at the current time into a target RB allocation model to output an RB allocation action of the network device at the current time.

[0256] The optional implementation of step S3402 can refer to the optional implementation of step S2105 in FIG. 2A and other associated parts in the embodiments involved in FIG. 2A, which will not be repeated here.

[0257] Step S3403: performing the RB allocation action to allocate resources to the ground signal device accessing the network device.

[0258] The optional implementation of step S3403 can refer to the optional implementation of steps S2106 and S2107 in FIG. 2A and other associated parts in the embodiments involved in FIG. 2A, which will not be repeated here.

[0259] In step S3404, the environment state at the current moment is input into the target power control model, and a power control action of the network device is output.

[0260] The optional implementation of step S3404 can refer to the optional implementation of step S2108 in FIG. 2A and other associated parts in the embodiments involved in FIG. 2A, which will not be repeated here.

[0261] In step S3405, the power control action is performed to adjust the power of the network device.

[0262] The optional implementation of step S3405 can refer to the optional implementation of step S2109 in FIG. 2A and other associated parts in the embodiments involved in FIG. 2A, which will not be repeated here.

[0263] The information processing method involved in the embodiments of the present disclosure can include at least one of steps S3401-S3405. For example, step S3402 can be implemented as an independent embodiment, step S3403 can be implemented as an independent embodiment, steps S3402+S3403 can be implemented as an independent embodiment, steps S3404+S3405 can be implemented as an independent embodiment, but are not limited thereto.

[0264] In the present embodiment or example, each step can be independently combined or exchanged in order without contradiction, and the optional mode or optional example can be combined with any step of other embodiments or other examples.

[0265] FIG. 3E is a flow diagram of an information processing method according to an embodiment of the present disclosure. As shown in FIG. 3E, the embodiments of the present disclosure involve an information processing method, and the method includes:

[0266] In step S3501, the environment state of the network device at the current moment is determined.

[0267] The optional implementation of step S3501 can refer to the optional implementation of step S2104 in FIG. 2A and other associated parts in the embodiments involved in FIG. 2A, which will not be repeated here.

[0268] In step S3502, the environment state at the current moment is input into the target DRL model to output the action of the network device at the current moment.

[0269] The optional implementation of step S3502 can refer to the optional implementation of steps S2106 and S2108 in FIG. 2A and other associated parts in the embodiments related to FIG. 2A, which will not be described here.

[0270] In step S3503, an action at the current time is performed to perform at least one of resource allocation and power control.

[0271] The optional implementation of step S3503 can refer to the optional implementation of steps S2107 and S2109 in FIG. 2A and other associated parts in the embodiments related to FIG. 2A, which will not be described here.

[0272] In the embodiments or examples, each step can be independent, arbitrarily combined or exchanged in sequence, the optional mode or optional example can be arbitrarily combined, and can be arbitrarily combined with any step of other embodiments or other examples.

[0273] The following is an exemplary introduction to the above method.

[0274] In the case of the fusion of the ground station of the satellite network and the ground cellular network, as shown in FIG. 4A, a joint framework of resource allocation and power control based on reinforcement learning processing is proposed in this example, which can learn the optimal strategy through frequent interaction with the time-varying environment to make adaptive decisions. Generally speaking, the problem modeling in DRL is based on a Markov decision process (MDP) with state space, action space, transition probability and immediate reward.

[0275] As shown in FIG. 4A, the actual environment is abstracted into Environment. Since the characteristics of RB allocation and power control are different, RB allocation is taken as a neural network (NN) and power control is taken as another neural network NN. The two neural networks learn respectively to form a joint framework of resource allocation and power control.

[0276] Action: Each agent takes its own action to decide the allocation of the corresponding RB and the subsequent power control according to the monitored local state information. State: At time t, the agent decides the action according to the observed state state t Take action, then the environment will transition to the next state state t+1 And feedback a reward reward t+1 The state must contain useful information to enable the agent to make appropriate decisions. Reward: The agent obtains an immediate reward reward t after performing the action t+1, which is an incentive for the agent to train an optimal policy that can choose appropriate actions to achieve the pre-specified goal.

[0277] Output the environmental state at time t from the environmental space The RB allocation model in the input agent outputs the RB allocation action (a m (t)), the environment space is based on the RB allocation action at time t (a m (t)), output the reward at time t+1 (r m (t+1)), further, a m (t), r m (t+1) and It is stored as experience data in the experience data set corresponding to the RB allocation model.

[0278] Output the environmental state at time t from the environmental space The power control model in the input agent outputs the power control action (p m (t)), the power control action of the environment space based on time t (p m (t)), output the reward at time t+1 (r m (t+1)), further, p m (t), r m (t+1) and The empirical data is stored in the empirical data set corresponding to the power control model.

[0279] Because RB allocation actions are discrete, while power control actions are continuous, this example combines two neural networks in the DQN and PPO frameworks. In this example, a joint optimization DRL framework is designed, and the agent uses the DQN algorithm to train the RB allocation model. The DQN algorithm, a combination of deep learning and Q-learning, is a value-based DRL algorithm suitable for training or optimizing models with discrete actions and large state spaces. Because the power control action values ​​are continuous, the agent uses the PPO algorithm to train the power control model. The PPO algorithm is an ideal choice for solving this type of problem and is an improvement over the AC algorithm, combining the advantages of both value-based and policy-based DRL algorithms.

[0280] In order to improve the training efficiency, a centralized training mechanism can be adopted for RB allocation and power control based on the agent. Through the interaction between the RB allocation model and power control model in the agent and the environment space, empirical data (that is, sample data) can be collected.

[0281] As shown in FIG. 4B, based on the two types of experience data obtained as shown in FIG. 4A, the RB allocation model and the power control model are jointly trained based on the experience data.

[0282] In some implementations, a first experience data is obtained from an experience data set corresponding to the RB allocation model, and the RB model is trained based on the first experience data according to the DQN algorithm. The first experience data includes: a state at time t, an RB allocation action (a m (t)) at time t, a reward (r m (t+1)) at time t+1, and a state

[0283] In some implementations, a second experience data is obtained from an experience data set corresponding to the power control allocation model, and the power control model is trained based on the second experience data according to the PPO algorithm. The second experience data includes: a state at time t, a power control action (p m (t)) at time t, a reward (r m (t+1)) at time t+1, and a state

[0284] In the training process, the loss functions of the RB allocation model and the power control model are obtained respectively, and the model parameters are adjusted based on the respective loss functions until the training end condition is met, to obtain the final target RB allocation model and the target power control model.

[0285] For a LEO satellite-ground communication network, the joint framework of DRL-based resource allocation and power control in the embodiments can maximize the resource utilization of the system.

[0286] The embodiments of the present disclosure also propose an apparatus for implementing any of the above methods, for example, an apparatus including units or modules for implementing each step performed by a terminal in any of the above methods. For another example, another apparatus is also proposed, including units or modules for implementing each step performed by a network device (such as an access network device, a core network function node, a core network device, etc.) in any of the above methods.

[0287] It should be understood that the division of each unit or module in the above apparatus is only a logical function division, and all or part of them can be integrated into a physical entity or physically separated in actual implementation. In addition, the units or modules in the apparatus can be implemented in the form of processor calling software: for example, the apparatus includes a processor, the processor is connected with a memory, the memory stores instructions, and the processor calls the instructions stored in the memory to realize any of the above methods or realize the functions of each unit or module of the above apparatus, wherein the processor is a general processor such as a central processing unit (CPU) or a microprocessor, and the memory is a memory in the apparatus or a memory outside the apparatus. Alternatively, the units or modules in the apparatus can be implemented in the form of hardware circuit, and the functions of part or all of the units or modules can be realized by the design of hardware circuit. The above hardware circuit can be understood as one or more processors; for example, in one implementation, the above hardware circuit is an application-specific integrated circuit (ASIC), and the functions of part or all of the units or modules are realized by the design of the logical relationship of elements in the circuit; for another example, in another implementation, the above hardware circuit is a programmable logic device (PLD), and a field programmable gate array (FPGA) is taken as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by a configuration file, so as to realize the functions of part or all of the above units or modules. All units or modules of the above apparatus can be all implemented in the form of processor calling software, or all implemented in the form of hardware circuit, or part implemented in the form of processor calling software and the remaining part implemented in the form of hardware circuit.

[0288] In the embodiments of the present disclosure, the processor is a circuit with signal processing capability. In one implementation, the processor can be a circuit with instruction reading and running capability, such as a central processing unit (CPU), a microprocessor, a graphics processing unit (GPU) (which can be understood as a microprocessor), a digital signal processor (DSP), or the like. In another implementation, the processor can implement certain functions through a logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or can be reconfigured. For example, the processor is a hardware circuit implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as an FPGA. In the reconfigurable hardware circuit, the processor loads a configuration document to implement the configuration of the hardware circuit. It can be understood that the processor loads instructions to implement the functions of the above part or all units or modules. In addition, the hardware circuit can also be designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), or the like.

[0289] FIG. 5 is a structural schematic diagram of a communication device according to an embodiment of the present disclosure. As shown in FIG. 5, the communication device 5100 can include at least one of a transceiver module 5101, a processing module 5102, and the like.

[0290] In some embodiments, the communication device 5100 is a network device. The processing module is configured to determine an environment state of an environment in which the network device is located at a current time; input the environment state of the current time into a target deep reinforcement learning (DRL) model to output an action of the network device in the current time; and perform the action in the current time to perform at least one of resource allocation and power control.

[0291] Optionally, the transceiver module and the processing module are configured to perform at least one of the communication steps, such as transmitting and / or receiving, performed by the network device in any of the above methods, which will not be described herein again.

[0292] In some embodiments, the transceiver module can include a transmitting module and / or a receiving module, which can be separate or integrated together. Optionally, the transceiver module can be replaced by a transceiver.

[0293] In some embodiments, the processing module can be one module, or can include multiple sub-modules. Optionally, the multiple sub-modules perform all or part of the steps required to be performed by the processing module, respectively. Optionally, the processing module can be mutually replaced with the processor.

[0294] FIG. 6A is a structural schematic diagram of a communication device 6100 according to an embodiment of the present disclosure. The communication device 6100 can be a first communication device (e.g., a core network device, etc.), a second communication device (e.g., an access network device, a user equipment, a core network device, etc.), a chip, a chip system, or a processor supporting the first communication device to implement any of the above methods, or a chip, a chip system, or a processor supporting the second communication device to implement any of the above methods. The communication device 6100 can be used to implement the methods described in the above method embodiments, and details can be referred to the descriptions in the above method embodiments.

[0295] As shown in FIG. 6A, the communication device 6100 includes one or more processors 6101. The processor 6101 can be a general-purpose processor or a special-purpose processor, etc., for example, a baseband processor or a central processing unit. The baseband processor can be used to process communication protocols and communication data, and the central processing unit can be used to control the communication device (e.g., a base station, a baseband chip, a terminal device, a terminal device chip, a DU or a CU, etc.), execute programs, and process data of the programs. Optionally, the communication device 6100 is configured to perform any of the above methods. Optionally, the one or more processors 6101 are configured to invoke instructions to cause the communication device 6100 to perform any of the above methods.

[0296] In some embodiments, the communication device 6100 further includes one or more transceivers 6102. When the communication device 6100 includes the one or more transceivers 6102, the transceiver 6102 performs at least one of the communication steps such as transmitting and / or receiving in the above methods, and the processor 6101 performs other steps. In optional embodiments, the transceiver can include a receiver and / or a transmitter, which can be separate or integrated together. Optionally, the terms of transceiver, transceiving unit, transceiver, transceiving circuit, interface circuit, interface, etc. can be mutually replaced, and the terms of transmitter, transmitting unit, transmitter, transmitting circuit, etc. can be mutually replaced, and the terms of receiver, receiving unit, receiver, receiving circuit, etc. can be mutually replaced.

[0297] In some embodiments, the communication device 6100 further includes one or more memories 6103 for storing data. Alternatively, all or part of the memories 6103 can be external to the communication device 6100. In optional embodiments, the communication device 6100 can include one or more interface circuits 6104. Optionally, the interface circuit 6104 is connected to the memory 6102, and the interface circuit 6104 can be used to receive data from the memory 6102 or other devices, and can be used to send data to the memory 6102 or other devices. For example, the interface circuit 6104 can read data stored in the memory 6102 and send the data to the processor 6101.

[0298] The communication device 6100 described in the above embodiments can be a network device or a terminal, but the scope of the communication device 6100 described in the present disclosure is not limited thereto, and the structure of the communication device 6100 can not be limited by Figure 6A. The communication device can be a standalone device or can be part of a larger device. For example, the communication device can be: 1) a standalone integrated circuit (IC), or a chip, or a chip system or subsystem; (2) a set of one or more ICs, which can optionally include a storage component for storing data, programs; (3) an ASIC, such as a modem; (4) a module that can be embedded in other devices; (5) a receiver, a terminal device, a smart terminal device, a cellular phone, a wireless device, a handset, a mobile unit, a vehicle-mounted device, a network device, a cloud device, an artificial intelligence device, etc.; (6) other devices, etc.

[0299] Figure 6B is a structural schematic diagram of a chip 6200 according to an embodiment of the present disclosure. For the case where the communication device 6100 is a chip or a chip system, the structural schematic diagram of the chip 6200 shown in Figure 6B can be referred to, but is not limited thereto.

[0300] The chip 6200 includes one or more processors 6201. The chip 6200 is configured to execute any of the above methods.

[0301] In some embodiments, the chip 6200 further includes one or more interface circuits 6202. Optionally, the terms interface circuit, interface, transceiver pin, etc. can be replaced by each other. In some embodiments, the chip 6200 further includes one or more memories 6203 for storing data. Optionally, all or part of the memories 6203 can be external to the chip 6200. Optionally, the interface circuit 6202 is connected to the memory 6203, and the interface circuit 6202 can be used to receive data from the memory 6203 or other devices, and the interface circuit 6202 can be used to send data to the memory 6203 or other devices. For example, the interface circuit 6202 can read data stored in the memory 6203 and send the data to the processor 6201.

[0302] In some embodiments, the interface circuit 6202 performs at least one of the communication steps (e.g., transmitting, receiving, etc.) in the above-described methods, which will not be repeated here. The interface circuit 6202 performing a communication step in the above-described methods refers to, for example, the interface circuit 6202 performing data interaction between the processor 6201, the chip 6200, the memory 6203, or a transceiver device. In some embodiments, the processor 6201 performs other steps.

[0303] The various modules and / or devices described in each of the embodiments of the virtual device, the physical device, the chip, etc. can be combined or separated according to the circumstances. Alternatively, part or all of the steps can also be performed by a plurality of modules and / or devices in cooperation, which is not limited here.

[0304] The disclosure further proposes a storage medium having instructions stored thereon, which, when executed on the communication device 6100, causes the communication device 6100 to perform any of the above methods. Alternatively, the storage medium is an electronic storage medium. Alternatively, the storage medium is a computer-readable storage medium, but is not limited to this, and it can also be a storage medium readable by other devices. Alternatively, the storage medium can be a non-transitory storage medium, but is not limited to this, and it can also be a transitory storage medium.

[0305] The disclosure further proposes a program product, which, when executed by the communication device 6100, causes the communication device 6100 to perform any of the above methods. Alternatively, the program product is a computer program product.

[0306] The disclosure further proposes a computer program, which, when executed on a computer, causes the computer to perform any of the above methods.

[0307] In the embodiments described above, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer programs. When the computer programs are loaded on a computer and executed, all or part of the processes or functions described in the embodiments of the present disclosure are produced. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer programs can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer programs can be transferred from one website, computer, server, or data center to another website, computer, server, or data center through a wired (such as a coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. that includes one or more available media sets. The available media can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a high-density digital video disc (DVD)), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0308] Those skilled in the art can appreciate that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present disclosure.

[0309] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0310] The above is merely a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present disclosure, which should be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. An information processing method characterized by comprising: The method is suitable for a network device, and comprises: determining an environment state of an environment in which the network device is located at a current time; inputting the environment state at the current time into a target deep reinforcement learning (DRL) model to output an action of the network device at the current time; performing the action at the current time to perform at least one of resource allocation and power control.

2. The method of claim 1, wherein, The target DRL model is a target resource block (RB) allocation model, and the inputting of the environment state at the current time into the target DRL model to output the action at the current time comprises: inputting the environment state at the current time into the target RB allocation model to output an RB allocation action of the network device at the current time.

3. The method of claim 2, wherein, The performing of the action at the current time comprises: performing the RB allocation action to allocate resources to a ground communication device accessing the network device.

4. The method of claim 3, wherein, After the performing of the RB allocation action to allocate resources to the ground communication device accessing the network device, the method further comprises: performing communication transmission with the ground communication device based on the allocated resources.

5. The method of claim 1, wherein, The target DRL model is a target power control model, and the inputting of the environment state at the current time into the target DRL model to output the action at the current time of the network device comprises: inputting the environment state at the current time into the target power control model to output a power control action of the network device at the current time.

6. The method of claim 5, wherein, The performing of the action at the current time comprises: performing the power control action to adjust power of the network device.

7. The method of claim 1, wherein, The method further comprises: based on interaction of an agent with an environment space, constructing a data replay pool, the environment space comprising environment states of the environment in which the network device is located collected at multiple different times; obtaining sample data from the data replay pool; training a DRL model based on the sample data to obtain the target DRL model.

8. The method of claim 7, wherein, The sample data comprises: an environment state at time t; an action at time t; an environment state at time t+1 after the action at time t is performed; an immediate reward at time t+1 fed back after the action at time t is performed.

9. The method of claim 7, wherein, The constructing of the data replay pool based on the interaction of the agent with the environment space comprises: inputting the environment state at time t in the environment space into a DRL model in the agent to output the action at time t; feeding back the action at time t to the environment space to perform to obtain the immediate reward at time t+1; taking the environment state at time t, the action at time t, the immediate reward at time t+1, and the environment state at time t+1 as the sample data; obtaining the data replay pool based on the sample data corresponding to multiple times. The method further comprises:

10. The method of claim 7, wherein, based on interaction of a first RB allocation model in an agent with the environment space, constructing a first data replay pool; obtaining first sample data from the first data replay pool; training a second RB allocation model based on the first sample data using a deep Q network (DQN) algorithm to obtain the target RB allocation model. The first sample data comprises:

11. The method of claim 10, wherein, an environment state at time t; an RB allocation action at time t. ​ an environment state at a t+1 moment after the RB allocation action at the t moment is performed; an immediate reward at the t+1 moment after the RB allocation action at the t moment is performed is fed back.

12. The method of claim 7, wherein, The method further includes: constructing a second data playback pool based on interactions of a first power control model in an agent with the environment space; obtaining second sample data from the second data playback pool; training a second power control model based on the second sample data using a proximal policy optimization (PPO) algorithm to obtain the target power control model.

13. The method of claim 12, wherein, The second sample data includes: an environment state at a t moment; a power control action at the t moment; an environment state at a t+1 moment after the power control action at the t moment is performed; an immediate reward at the t+1 moment after the power control action at the t moment is performed is fed back.

14. The method of claim 10 or 12, wherein, The method further includes: training a second RB allocation model and a second power control model based on the agent and the environment space to obtain a target RB allocation model and a target power control model.

15. The method of claim 14, wherein, The inputting of the environment state at the current moment into the target deep reinforcement learning (DRL) model to output an action of the network device at the current moment includes: inputting the environment state at the current moment into the target RB allocation model to output an RB allocation action of the network device at the current moment; inputting the environment state at the current moment into the target power control model to output a power control action of the network device at the current moment.

16. A communication device, characterized by include: a processing module configured to determine an environment state of an environment at a current moment at which a communication device is located; inputting the environment state at the current moment into a target DRL model to output an action of the communication device at the current moment; performing the action at the current moment to perform at least one of resource allocation and power control.

17. A communication device, characterized by include: one or more processors; The communication device is configured to perform the information processing method in any one of claims 1-15.

18. A storage medium, the storage medium storing instructions, wherein, The instructions, when executed on the communication device, cause the communication device to perform the information processing method in any one of claims 1-15.

19. A program product, which, when executed on a communication device, causes the communication device to perform the information processing method in any one of claims 1-15.

Images