Multi-agent information processing method and related apparatus

Abstract

Disclosed in the present application are a multi-agent information processing method and a related apparatus. The method comprises: acquiring first hidden state information of first observation information, wherein the first observation information is local observation information of a first agent; acquiring aggregated information, wherein the aggregated information is obtained by means of performing aggregation on first information of at least one second agent, the first information of each second agent is obtained by means of performing encoding on second observation information, and the second observation information is local observation information of each second agent; and on the basis of the first hidden state information and the aggregated information, determining an action to be executed. By means of the embodiments of the present application, encoding and aggregation can be performed on local observation information of an agent, such that when aggregated information is used as an input of an action policy decision, the consumption of computing resources and transmission resources can be reduced.

Description

A multi-agent information processing method and related apparatus

[0001] This application claims priority to Chinese Patent Application No. 202410686478.8, filed on May 29, 2024, entitled "A Multi-Agent Information Processing Method and Related Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the fields of communication and artificial intelligence technologies, and in particular to a multi-agent information processing method and related apparatus. Background Technology

[0003] The close coupling of communication networks and artificial intelligence (AI) technologies enables wireless networks to possess self-organizing, self-optimizing, and scalable capabilities. Deep reinforcement learning (DRL) is currently widely used to solve various sequential decision problems, providing a model-free intelligent solution for communication network optimization. Multi-agent reinforcement learning (MARL), in particular, is a tool for solving multi-user resource allocation problems in a distributed and adaptive manner. In existing MARL models, agents typically make action decisions based on local observations of the communication network. In multi-agent collaborative scenarios, agents exchange local observation information to better identify the global environment state. The amount of data from these local observations is relatively large, and using it as input for action policy decisions increases the computational resource consumption of the agents. Summary of the Invention

[0004] This application provides a multi-agent information processing method and related apparatus that can encode and aggregate local observation information of agents, thereby reducing the consumption of computing and transmission resources when used as input for action strategy decision-making.

[0005] In a first aspect, embodiments of this application provide a multi-agent information processing method, applied to a first agent, the method comprising:

[0006] Acquire the first hidden state information of the first observation information; the first observation information is the local observation information of the first agent.

[0007] Acquire aggregated information; aggregated information is obtained by aggregating the first information of at least one second agent, and the first information of each second agent is obtained by encoding the second observation information; the second observation information is the local observation information of each second agent.

[0008] The action to be performed is determined based on the first hidden state information and the aggregate information.

[0009] As can be seen, in this embodiment of the application, the first agent can obtain aggregated information of the first information of at least one second agent. Since the first information is obtained by encoding the local observation information of the second agent, its data volume is smaller than that of the original observation information, and its aggregated information is also relatively small. The first agent uses the hidden state information of the local observation information and the aggregated information as inputs for action policy decision-making, which relatively reduces the amount of input data for local prediction, thereby helping to reduce the consumption of computing resources.

[0010] In one possible implementation, obtaining aggregated information includes:

[0011] Receive collaborative messages from at least one second agent; each collaborative message from a second agent includes first information about each second agent.

[0012] The weights of the first information of at least one second agent are determined based on the first hidden state information.

[0013] Based on the weights of the first information of at least one second agent, the first information of at least one second agent is aggregated to obtain aggregated information.

[0014] In this implementation, the second agent can send the generated first information to the first agent via a cooperative message. Since the first information is encoded and relatively small compared to the original observation information, it helps to save transmission overhead and reduce the consumption of transmission resources. Upon receiving a cooperative message from at least one second agent, the first agent determines the weight of the first information of at least one second agent based on the first hidden state information, and aggregates the first information of at least one second agent using this weight. This allows the aggregated information to be used as input for local action decisions instead of the original observation information, thus reducing the consumption of local computing resources.

[0015] In one possible implementation, the collaborative message of each second agent also includes signature information of each second agent; the signature information of each second agent is generated based on the second hidden state information; the second hidden state information is the hidden state information of the second observation information of each second agent.

[0016] Determining the weights of the first information of at least one second agent based on the first hidden state information includes:

[0017] Retrieve query information; the query information is generated based on the first hidden state information;

[0018] The weight of the first information of each second agent is calculated by using the query information and the signature information of each second agent.

[0019] In this implementation, the first agent performs an inner product of the generated query information and the signature information of each second agent to obtain the weight of the first information for each second agent. A larger weight indicates a stronger correlation between the observations of the two agents, suggesting an overlap between their state and action spaces and a higher degree of mutual influence. Performing an inner product of the weight and the corresponding first information amplifies the more correlated first information; therefore, aggregating information helps improve the accuracy of the first agent's action decisions.

[0020] In one possible implementation, the weights of the first information for each second agent satisfy:

[0021] Among them, a i This represents a vector composed of the weights of the first information of each second agent. This represents the transpose of the query information, k1,…,k i-1 ,k i+1 ,…,k U This represents the signature information of at least one second agent, where U represents the total number of agents.

[0022] In this implementation, the weights corresponding to at least one second agent constitute the weight vector of the first agent, which facilitates the subsequent aggregation of the first information.

[0023] In one possible implementation, the method further includes, before obtaining the aggregated information:

[0024] Query information is generated based on the first hidden state information;

[0025] Send query information to network devices;

[0026] Obtain aggregated information, including:

[0027] The network device receives aggregated information sent by the network device. The aggregated information is obtained by the network device aggregating the first information of at least one second agent based on the query information and the signature information of at least one second agent. The signature information of each second agent is generated based on the second hidden state information. The second hidden state information is the hidden state information of the second observation information of each second agent.

[0028] In this implementation, the first intelligent agent receives the aggregated information sent by the network device. That is, the information aggregation operation is implemented by the network device, which helps to reduce the interaction of collaborative messages between intelligent agents. Each intelligent agent only needs to send the generated information to the network device, thereby reducing the redundancy of information in the network.

[0029] In one possible implementation, receiving a collaborative message from at least one second agent includes:

[0030] Receive collaborative messages from at least one second agent forwarded by network devices.

[0031] In this implementation, the first and second intelligent agents can exchange collaborative information through network devices, so that each intelligent agent can obtain the observation information of other intelligent agents and thus obtain global observation information.

[0032] In one possible implementation, after obtaining the first hidden state information of the first observation information, the method further includes:

[0033] Generate the second information and signature information of the first intelligent agent based on the first hidden state information;

[0034] Send a collaboration message of the first intelligent agent to the network device; the collaboration message of the first intelligent agent includes the second information of the first intelligent agent and the signature information of the first intelligent agent.

[0035] In this implementation, the first intelligent agent can send the second information generated by encoding the first hidden state information and the signature information generated based on the first hidden state information to the network device, so that the network device can forward them to at least one second intelligent agent, thereby facilitating information aggregation by at least one second intelligent agent. Alternatively, the network device may not forward the second information and signature information of the first intelligent agent, but can still perform information aggregation for at least one second intelligent agent based on the second information and signature information of the first intelligent agent.

[0036] In one possible implementation, the action to be performed is determined based on the first hidden state information and the aggregation information, including:

[0037] The first hidden state information is used as the input to the policy network, and the aggregated information is used as the bias of the policy network. The action to be executed is then predicted by the policy network.

[0038] Alternatively, the first hidden state information and the aggregation information can be used as inputs to the policy network, which can then predict the actions to be performed.

[0039] In this implementation, the first agent uses the aggregated information as a bias in the policy network without increasing the dimensionality of the network input, which helps reduce the complexity of the policy space and action decisions. However, by adding aggregated information as a new input to the policy network, the first agent increases the dimensionality of the network input, effectively broadening its local observation space and enabling it to learn more information from the environment.

[0040] Secondly, embodiments of this application provide a multi-agent information processing device applied to a first agent. The device includes a processing unit; the processing unit is used for:

[0041] Acquire the first hidden state information of the first observation information; the first observation information is the local observation information of the first agent.

[0042] Acquire aggregated information; aggregated information is obtained by aggregating the first information of at least one second agent, and the first information of each second agent is obtained by encoding the second observation information; the second observation information is the local observation information of each second agent.

[0043] The action to be performed is determined based on the first hidden state information and the aggregate information.

[0044] In one possible implementation, the device further includes a transceiver unit; in acquiring aggregated information, the processing unit is specifically used for:

[0045] The transceiver unit receives collaborative messages from at least one second agent; each collaborative message from a second agent includes first information about that second agent.

[0046] The weights of the first information of at least one second agent are determined based on the first hidden state information.

[0047] Based on the weights of the first information of at least one second agent, the first information of at least one second agent is aggregated to obtain aggregated information.

[0048] In one possible implementation, the collaborative message of each second agent also includes signature information of each second agent; the signature information of each second agent is generated based on the second hidden state information; the second hidden state information is the hidden state information of the second observation information of each second agent.

[0049] In determining the weights of the first information of at least one second agent based on the first hidden state information, the processing unit is specifically used for:

[0050] Retrieve query information; the query information is generated based on the first hidden state information;

[0051] The weight of the first information of each second agent is calculated by using the query information and the signature information of each second agent.

[0052] In one possible implementation, the weights of the first information for each second agent satisfy:

[0053] Among them, a i This represents a vector composed of the weights of the first information of each second agent. This represents the transpose of the query information, k1,…,k i-1 ,k i+1 ,…,k U This represents the signature information of at least one second agent, where U represents the total number of agents.

[0054] In one possible implementation, the device further includes a transceiver unit; the processing unit is further configured to: generate query information based on the first hidden state information;

[0055] The transceiver unit is used to send query information to network devices;

[0056] In acquiring aggregated information, the processing unit is specifically used for:

[0057] The network device receives aggregated information sent by the transceiver unit. The aggregated information is obtained by the network device by aggregating the first information of at least one second agent based on the query information and the signature information of at least one second agent. The signature information of each second agent is generated based on the second hidden state information. The second hidden state information is the hidden state information of the second observation information of each second agent.

[0058] In one possible implementation, the transceiver unit is specifically configured to: receive cooperative messages from at least one second agent.

[0059] Receive collaborative messages from at least one second agent forwarded by network devices.

[0060] In one possible implementation, the processing unit is further configured to: generate second information of the first agent and signature information of the first agent based on the first hidden state information;

[0061] The transceiver unit is also used to: send a cooperation message of the first intelligent agent to the network device; the cooperation message of the first intelligent agent includes the second information of the first intelligent agent and the signature information of the first intelligent agent.

[0062] In one possible implementation, the processing unit is specifically used to determine the action to be performed based on the first hidden state information and the aggregation information:

[0063] The first hidden state information is used as the input to the policy network, and the aggregated information is used as the bias of the policy network. The action to be executed is then predicted by the policy network.

[0064] Alternatively, the first hidden state information and the aggregation information can be used as inputs to the policy network, which can then predict the actions to be performed.

[0065] It should be understood that since the method embodiments and the device embodiments are different presentations of the same technical concept, the content of the first aspect of the embodiments of this application should be adapted to the second aspect of the embodiments of this application simultaneously, and can achieve the same or similar beneficial effects, which will not be repeated here.

[0066] Thirdly, embodiments of this application provide a terminal device, including a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to, when executed by the processor, cooperate with the communication interface to implement the method in any of the embodiments of the first aspect described above.

[0067] Fourthly, embodiments of this application provide a chip, including: a processor, configured to call and run a computer program from a memory, causing a device on which the chip is installed to perform the method described in the first aspect above.

[0068] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program for execution by a device, wherein the computer program, when executed, implements the method as described in any of the embodiments of the first aspect above.

[0069] Sixthly, embodiments of this application provide a computer program product that, when run by a terminal device, causes the terminal device to perform the method as described in any of the embodiments of the first aspect above. Attached Figure Description

[0070] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.

[0071] Figure 1 is a schematic diagram of a system architecture;

[0072] Figure 2 is a flowchart illustrating an information processing method for multiple agents provided in an embodiment of this application;

[0073] Figure 3 is a schematic diagram of the internal modules of an intelligent agent provided in an embodiment of this application;

[0074] Figure 4 is a schematic diagram of obtaining aggregated information based on the interaction between intelligent agents according to an embodiment of this application;

[0075] Figure 5 is a schematic diagram of the input and output of an internal module of an intelligent agent provided in an embodiment of this application;

[0076] Figure 6 is a schematic diagram of an interactive acquisition of aggregated information based on network devices according to an embodiment of this application;

[0077] Figure 7 is a flowchart illustrating another multi-agent information processing method provided in an embodiment of this application;

[0078] Figure 8 is a schematic diagram of an aggregation module deployed in a network device according to an embodiment of this application;

[0079] Figure 9 is a schematic diagram of an experimental test result provided in an embodiment of this application;

[0080] Figure 10 is a schematic diagram of the structure of a multi-agent information processing device provided in an embodiment of this application;

[0081] Figure 11 is a schematic diagram of another multi-agent information processing device provided in an embodiment of this application;

[0082] Figure 12 is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0083] The terms "first," "second," "third," and "fourth," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0084] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0085] The terms “component,” “module,” “system,” etc., used in this specification are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, an application running on a terminal device and the terminal device can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0086] First, a brief introduction to the relevant terms and technical background used in this application will be provided to facilitate understanding by those skilled in the art.

[0087] (1) Orthogonal Frequency Division Multiplexing (OFDM);

[0088] (2) Base stations: gNodeB, gNB;

[0089] (3) User Equipment (UE);

[0090] (4) Peak to Average Power Ratio (PAPR)

[0091] (5) Demodulation Reference Signal (DMRS);

[0092] (6) Physical Uplink Shared Channel (PUSCH);

[0093] (7) Physical Uplink Control Channel (PUCCH);

[0094] (8) Physical Downlink Control Channel (PDCCH);

[0095] (9) Physical Downlink Shared Channel (PDSCH);

[0096] (10) Reference Signals: RS;

[0097] (11) New Radio: NR;

[0098] (12) User Equipment: UserEquipment, UE;

[0099] (13) Radio Resource Control (RRC);

[0100] (14) Core Network: CN;

[0101] (15) Radio Access Network (RAN);

[0102] (16) Long Term Evolution (LTE) is a long-term evolution of the Universal Mobile Telecommunications System (UMTS) technical standard developed by the 3rd Generation Partnership Project (3GPP).

[0103] (17) Enhance Mobile Broadband (eMBB) refers to further improvements in performance such as user experience based on existing mobile broadband service scenarios. This is also the application scenario that is closest to our daily lives.

[0104] (18) Ultra-reliable and low-latency communication (URLLC): URLLC is characterized by high reliability, low latency, and extremely high availability. It can be applied to various scenarios such as industrial applications and control, traffic safety and control, remote manufacturing, remote training, and remote surgery.

[0105] (19) Machine-type communication (MTC) refers to wireless communication across a wide range of fields involving sensors, actuators, physical objects, and other devices not directly operated by humans. The machines can be various types of IoT devices, sensors, industrial automation equipment, etc. MTC features massive connectivity, low power consumption and low cost, low latency and high reliability, diverse application scenarios, and security and privacy protection.

[0106] (20) Narrow Band Internet of Things (NB-IoT): Features wide coverage, high connectivity, low data rate, low cost, low power consumption, and superior architecture. Application scenarios include: smart water meters, smart parking, smart pet tracking, smart bicycles, smart smoke detectors, smart toilets, smart vending machines, etc.

[0107] (21) Vehicle to everything is a key technology for future intelligent transportation systems. It enables communication between vehicles, between vehicles and base stations, and between base stations, thereby obtaining a series of traffic information such as real-time road conditions, road information, and pedestrian information, thereby improving driving safety, reducing congestion, improving traffic efficiency, and providing in-vehicle entertainment information.

[0108] Many current studies utilize Multi-Agent Relationships (MARL) in a centralized training and decentralized execution (CTDE) scheme to address the inherent non-steady-state problems of multi-agent systems. While global environmental information is provided during the training phase to guide multi-user cooperation, agents typically learn and generate actions based on their local observations. In complex scenarios where agents need to cooperate to adhere to resource constraints, agents with only local observation capabilities may become confused about the true environmental state, thus degrading performance. To better facilitate cooperative decision-making, introducing inter-agent communication into MARL has become a new and popular research direction, often referred to as MARL with spontaneous communication. In this research direction, agents exchange locally observed information by learning communication policies to better identify the global environmental state. However, in dynamic mobile communication networks, communication opportunities and resources are limited. Excessive raw observation information can excessively consume communication resources, while for a single agent, its computational resources are usually fixed. Using raw observation information as input to the local neural network or policy network often increases the agent's computational resource consumption.

[0109] To address the shortcomings of related technologies, this application provides a multi-agent information processing method. This method can be applied to high-order modulation in the evolution of 5G and subsequent 6G mobile networks, and can be implemented based on the system architecture shown in Figure 1. As shown in Figure 1, this system architecture includes at least terminal devices, base stations, and a core network. Terminal devices and base stations can act as transmitters and receivers to each other. Transmitters can send their own cooperative messages to receivers, forward cooperative messages from other terminal devices, or send aggregated messages generated based on cooperative messages from other terminal devices. Essentially, terminal devices obtain global observation information based on cooperative messages from other terminal devices to make local decisions.

[0110] For example, a terminal device can also be called a terminal, and can be a device with wireless transceiver capabilities. A terminal device can be a UE (User Equipment), where a UE includes a handheld device, in-vehicle device, wearable device, or computing device with wireless communication capabilities. For example, a UE can be a mobile phone, tablet computer, or computer with wireless transceiver capabilities. A terminal device can also be a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in industrial control, a wireless terminal in autonomous driving, a wireless terminal in telemedicine, a wireless terminal in a smart grid, a wireless terminal in a smart city, a wireless terminal in a smart home, and so on. In this application embodiment, a terminal device can be a device for implementing the functions of a terminal; it can also be a device that supports the terminal in implementing those functions, such as a chip system, which can be installed in the terminal. In this application embodiment, the chip system can be composed of chips, or it can include chips and other discrete components.

[0111] For example, a base station can be a device deployed in a wireless access network capable of wirelessly communicating with terminals. A base station can be a macro base station, micro base station, relay station, or access point, etc. For example, the base station involved in the embodiments of this application can be a 5G base station or an eNB (Evolved Node B) base station in LTE. The 5G base station can also be called a Transmission Reception Point (TRP) or a gNB (Next-Generation Node B) base station. In the embodiments of this application, the device used to implement the functions of the base station can be the base station itself; it can also be a device that supports the base station in implementing this function, such as a chip system, which can be installed in the base station. In the embodiments of this application, the technical solution provided by the embodiments of this application is described using the base station itself as an example of the device used to implement the functions of the base station. Optionally, in some deployments of base stations, the base station can be a central unit (CU) or a distributed unit (DU), etc. For example, operations or steps at the Radio Link Control (RLC) layer, Media Access Control (MAC) layer, and RRC layer can be performed by the CU, while operations or steps at the Physical (PHY) layer can be performed by the DU. In some other base station deployments, the CU can also be divided into a CU-control plane (CP) and a CU-user plane (UP), etc. In still other base station deployments, the base station can also be an antenna unit (RU). In still other base station deployments, the base station can also be an open radio access network (ORAN) architecture, etc. The embodiments of this application do not limit the deployment method of the base station. For example, when the base station is an ORAN architecture, the base station shown in the embodiments of this application can be an access network device in ORAN, or a module in the access network device, etc. In the ORAN architecture, the CU can also be called an open (O)-CU, the DU can also be called an O-DU, the CU-CP can also be called an O-CU-CP, the CU-UP can also be called an O-CU-UP, and the RU can also be called an O-RU.

[0112] The core network's main functions are to provide user connections, manage users, and carry out services, while also serving as an interface to external networks.

[0113] Based on the system architecture shown in Figure 1, the embodiments of this application are applicable to both low-frequency scenarios (sub 6G) and high-frequency scenarios (above 6G), such as terahertz and optical communication.

[0114] The technical solution provided in this application will be described in detail below with reference to specific implementation methods.

[0115] Please refer to Figure 2, which is a flowchart illustrating a multi-agent information processing method provided in an embodiment of this application. This method can be implemented based on the system architecture shown in Figure 1. As shown in Figure 2, the method includes steps 201-203:

[0116] 201: The first agent acquires the first hidden state information of the first observation information.

[0117] The first observation information refers to the local observation information of the first intelligent agent, that is, the observation values ​​obtained by the first intelligent agent from the learning environment. For example, in a wireless communication environment, the first intelligent agent can be a terminal device (such as a UE), and the first observation information can include the base stations that the terminal device has accessed, the bandwidth of communication with the base station, the power interference of the base station, the reference signal receiving power (RSRP) of the base station, and so on.

[0118] Please refer to Figure 3. Each agent includes an observation memory network, which can be a gated recurrent unit (GRU). It also includes a message generation network (MG), a message aggregation network (MA), and a policy network. For the first observation information O acquired by the first agent at the current time... i It is then input into the GRU, which uses the hidden state information h from the previous time step. i (t-1) and the first observation information O i Perform reasoning to generate the hidden state information h at the current moment. i (t), representing the first hidden state information. i indicates that the first agent is the i-th agent. The MG network, MA network, and policy network can be neural network models.

[0119] 202: The first intelligent agent acquires aggregated information.

[0120] In this embodiment of the application, the aggregated information is obtained by aggregating the first information of at least one second agent, and the first information of each second agent is obtained by encoding the second observation information; the second observation information is the local observation information of each second agent, that is, the first information can characterize the local observation information of the second agent.

[0121] For example, as shown in Figure 4, after obtaining the first hidden state information of the first observation information, the method further includes:

[0122] 401: The first intelligent agent generates the second information of the first intelligent agent and the signature information of the first intelligent agent based on the first hidden state information;

[0123] In this embodiment of the application, as shown in Figure 5, the MG network includes an encoder module and a signature key module. The encoder module takes the first hidden state information as input and encodes it into the second information of the first agent. The second information can be represented as: m i =f enc (h i (t))

[0124] Among them, f enc This represents the encoding function of the encoder module. GRU can also be viewed as an encoder, so the second information of the first agent can be seen as obtained by encoding the first observation information; that is, the second information can characterize the first observation information.

[0125] The `key` module is used to sign the first hidden state information to generate the signature information of the first agent. The signature information of the first agent can then be represented as: k i =f key (h i (t))

[0126] Among them, f key This represents the signature function of the key module.

[0127] 402: The first agent sends a cooperation message to at least one second agent;

[0128] The collaborative message of the first agent includes the second information of the first agent and the signature information of the first agent.

[0129] 403: At least one second agent sends a cooperation message to the first agent;

[0130] Each second agent's collaborative message includes its first information and its signature. Similar to the generation method of the first agent's signature, each second agent's signature is generated based on its second hidden state information. The second hidden state information is the hidden state information of each second agent's second observation information, and its acquisition method is the same as that of the first hidden state information.

[0131] Then, the first intelligent agent acquires aggregated information, including:

[0132] 404: The first agent receives a cooperative message from at least one second agent;

[0133] There is no time restriction between steps 402 and 404.

[0134] 405: The first agent determines the weights of at least one second agent's first information based on the first hidden state information;

[0135] In this embodiment, please refer to Figure 5. The MA network includes a Query module and a Merge module. The Query module takes the first hidden state information as input to generate query information for the first agent. The query information of the first agent can be represented as: q i =f query (h i (t))

[0136] Among them, f query This refers to the query information generation function of the Query module. The query information of the first agent can be generated before or after receiving a collaborative message from at least one second agent. If it is generated before, the first agent can directly obtain the pre-generated query information. This application does not limit the generation time.

[0137] The Merge module performs an inner product of the query information of the first agent and the first information of each second agent to obtain the weight of the first information of at least one second agent. This weight then satisfies the following:

[0138] Among them, a i This represents a vector composed of the weights of the first information of each second agent. This represents the transpose of the query information, k1,…,k i-1 ,k i+1 ,…,k U This represents the signature information of at least one second agent, where U represents the total number of agents.

[0139] In this implementation, the second agent can send the generated first information to the first agent via a cooperative message. Since the first information is encoded and relatively small compared to the original observation information, it helps save transmission overhead and reduce the consumption of transmission resources. Upon receiving a cooperative message from at least one second agent, the first agent determines the weight of the first information of at least one second agent based on the first hidden state information, and aggregates the first information of at least one second agent using this weight. This allows the aggregated information to be used as input for local action decisions instead of the original observation information, thus reducing the consumption of local computing resources. The weight of the first information of each second agent is obtained by performing an inner product with the generated query information and the signature information of each second agent. The weights corresponding to at least one second agent constitute the weight vector of the first agent, facilitating subsequent aggregation of the first information. A larger weight indicates a stronger correlation between the observations of the two agents, resulting in an overlap between the state space and action space, and a higher degree of mutual influence. Performing an inner product of the weight and the corresponding first information is equivalent to amplifying the more correlated first information; therefore, aggregating the information helps improve the accuracy of the first agent's action decisions.

[0140] 406: The first agent aggregates the first information of at least one second agent based on the weights of the first information of at least one second agent to obtain aggregated information.

[0141] In this embodiment of the application, the first intelligent agent has the first information m for all second intelligent agents. -i By performing a weighted summation, we obtain the aggregated information, which can then be represented as:

[0142] Among them, a i [j] represents the weight vector a i The j-th weight in m j This represents the first information of the second agent corresponding to the j-th weight.

[0143] For example, steps 402 and 403 above can also be implemented via a network device. As shown in Figure 6, after obtaining the first hidden state information of the first observation information, the method further includes:

[0144] 601: The first intelligent agent generates the second information of the first intelligent agent and the signature information of the first intelligent agent based on the first hidden state information;

[0145] 602: The first intelligent agent sends a cooperation message to the network device;

[0146] 603: A network device sends a cooperation message from a first agent to at least one second agent;

[0147] 604: At least one second agent sends a cooperation message to the network device;

[0148] 605: The network device sends a cooperation message from at least one second agent to the first agent;

[0149] 606: The first agent receives a cooperative message forwarded by a network device from at least one second agent;

[0150] 607: The first agent determines the weights of at least one second agent's first information based on the first hidden state information;

[0151] 608: The first agent aggregates the first information of at least one second agent based on the weights of the first information of at least one second agent to obtain aggregated information.

[0152] In this implementation, the first and second intelligent agents can exchange collaborative information through network devices, so that each intelligent agent can obtain the observation information of other intelligent agents and thus obtain global observation information.

[0153] 203: The first intelligent agent determines the action to be performed based on the first hidden state information and the aggregate information.

[0154] In this embodiment, the first hidden state information can represent local observation information, and the aggregated information can represent the observation information of at least one second agent. Therefore, the first hidden state information and the aggregated information are used as inputs to the policy network, enabling the policy network to learn global observation information. The policy network can be represented as: π(A u |O u ,m -u ) m -u =m1,…,m u-1 ,m u+1 ,…,m U ∈M -u

[0155] Where, m -u O represents the information received by the u-th agent from all other agents. u M represents the local observation information of the u-th agent. -u Let m represent the set of information of all other intelligent agents. u The dimension is k, A u Indicates that the u-th agent is based on O u and m -u The chosen action is the action that needs to be performed.

[0156] For example, the first agent can use the first hidden state information as input to the policy network and the aggregated information as the bias of the policy network to predict the action to be performed: Net i =Net i (h i (t))+Merge i

[0157] Among them, Net i This represents the output of the policy network. In this implementation, the first agent uses aggregated information as the bias of the policy network without increasing the dimensionality of the network input, which helps to reduce the complexity of the policy space and action decisions.

[0158] For example, the first agent uses the first hidden state information and the aggregated information as input to the policy network, and the policy network predicts the action to be performed: Net i =Net i (h i (t),Merge i )

[0159] In this implementation, the first agent uses aggregated information as a new input to the policy network, which increases the dimension of the network input. This is equivalent to broadening the local observation space of the first agent, enabling it to learn more information from the environment.

[0160] As can be seen, in this embodiment of the application, the first agent can obtain aggregated information of the first information of at least one second agent. Since the first information is obtained by encoding the local observation information of the second agent, its data volume is smaller than that of the original observation information, and its aggregated information is also relatively small. The first agent uses the hidden state information of the local observation information and the aggregated information as inputs for action policy decision-making, which relatively reduces the amount of input data for local prediction, thereby helping to reduce the consumption of computing resources.

[0161] Please refer to Figure 7, which is a flowchart illustrating another multi-agent information processing method provided in an embodiment of this application. This method can be implemented based on the system architecture shown in Figure 1. As shown in Figure 7, the method includes steps 701-707:

[0162] 701: The first agent acquires the first hidden state information of the first observation information;

[0163] 702: The first intelligent agent generates query information based on the first hidden state information;

[0164] 703: The first intelligent agent sends a query to the network device;

[0165] For example, as shown in Figure 8, the Query module of the MA network is deployed locally on the agent, and the Merge module of the MA network is deployed on the network device. For the first agent, the encoder module generates second information based on the first hidden state information, and the key module generates signature information based on the first hidden state information. For the second agent, the encoder module generates first information based on the second hidden state information, the key module generates signature information based on the second hidden state information, and the Query module generates query information based on the second hidden state information. The first agent sends a cooperation message to the network device, which includes the first agent's signature information and second information. The second agent sends a cooperation message and a query to the network device, which includes the second agent's signature information and the first information.

[0166] 704: The network device aggregates the first information of at least one second agent based on the query information of the first agent and the signature information of at least one second agent to obtain aggregated information;

[0167] In this embodiment, the network device aggregates the information generated by the encoder modules of other intelligent agents based on the query information of each intelligent agent and the signature information of other intelligent agents to obtain the aggregated information of each intelligent agent. The specific aggregation method can be referred to the relevant descriptions in steps 405 and 406 above.

[0168] 705: The network device sends aggregated information to the first intelligent agent;

[0169] Similarly, network devices send aggregated information to at least one second agent.

[0170] 706: The first intelligent agent receives aggregated information sent by the network device;

[0171] 707: The first agent determines the action to be performed based on the first hidden state information and the aggregate information.

[0172] The implementation methods of steps 701 and 707 can be referred to the relevant descriptions in steps 201 and 203.

[0173] In this implementation, the aggregation function of the MA network is deployed on the network device, and the information aggregation operation is implemented by the network device. This helps to reduce the interaction of collaborative messages between agents. Each agent only needs to send the generated information to the network device, thereby reducing the redundancy of information in the network.

[0174] The embodiments of this application can be applied to scenarios such as wireless network access. For example, a UE can make access strategy decisions based on local observation information and observation information from other UEs, such as power control, sub-channel selection, and interference coordination. Those skilled in the art, taking UE decision-making for interference coordination as an example, conducted experimental tests on the scheme provided in this application, the Independent Q-Learning (IQL) method, and the Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (QMIX) method. The test results are shown in Figure 9. In Figure 9, the horizontal axis represents the number of data iteration rounds, and the vertical axis represents the negative of the data transmission delay. As can be seen from Figure 9, the scheme provided in this application, through the encoding, transmission, and aggregation of newly added observation information, significantly outperforms the other two schemes in terms of performance gain. It also improves the accuracy of agent decision-making while reducing user data packet transmission delay.

[0175] The methods of the embodiments of this application have been described above, and the apparatus of the embodiments of this application is provided below.

[0176] Please refer to Figure 10, which is a schematic diagram of the structure of a multi-agent information processing device provided in an embodiment of this application. As shown in Figure 10, the device includes at least a processing unit 1001; wherein, the processing unit 1001 is used for:

[0177] Acquire the first hidden state information of the first observation information; the first observation information is the local observation information of the first agent.

[0178] Acquire aggregated information; aggregated information is obtained by aggregating the first information of at least one second agent, and the first information of each second agent is obtained by encoding the second observation information; the second observation information is the local observation information of each second agent.

[0179] The action to be performed is determined based on the first hidden state information and the aggregate information.

[0180] As can be seen, in the device shown in Figure 10, aggregated information of at least one second agent's first information can be obtained. Since the first information is obtained by encoding the local observation information of the second agent, its data volume is smaller than that of the original observation information, and its aggregated information is also relatively small. The device uses the hidden state information of the local observation information and the aggregated information as inputs for action policy decision-making, which relatively reduces the amount of input data for local prediction, thereby helping to reduce the consumption of computing resources.

[0181] In one possible implementation, as shown in Figure 11, the device further includes a transceiver unit 1002; in terms of acquiring aggregated information, the processing unit 1001 is specifically used for:

[0182] The transceiver unit 1002 receives cooperative messages from at least one second agent; each cooperative message from the second agent includes first information about the second agent.

[0183] The weights of the first information of at least one second agent are determined based on the first hidden state information.

[0184] Based on the weights of the first information of at least one second agent, the first information of at least one second agent is aggregated to obtain aggregated information.

[0185] In one possible implementation, the collaborative message of each second agent also includes signature information of each second agent; the signature information of each second agent is generated based on the second hidden state information; the second hidden state information is the hidden state information of the second observation information of each second agent.

[0186] In determining the weights of the first information of at least one second agent based on the first hidden state information, the processing unit 1001 is specifically configured to:

[0187] Retrieve query information; the query information is generated based on the first hidden state information;

[0188] The weight of the first information of each second agent is calculated by using the query information and the signature information of each second agent.

[0189] In one possible implementation, the weights of the first information for each second agent satisfy:

[0190] Among them, a i This represents a vector composed of the weights of the first information of each second agent. This represents the transpose of the query information, k1,…,k i-1 ,k i+1 ,…,k U This represents the signature information of at least one second agent, where U represents the total number of agents.

[0191] In one possible implementation, the processing unit 1001 is further configured to: generate query information based on the first hidden state information;

[0192] The transceiver unit 1002 is used to send query information to network devices;

[0193] In acquiring aggregated information, the processing unit 1001 is specifically used for:

[0194] The network device receives aggregated information sent by the transceiver unit. The aggregated information is obtained by the network device by aggregating the first information of at least one second agent based on the query information and the signature information of at least one second agent. The signature information of each second agent is generated based on the second hidden state information. The second hidden state information is the hidden state information of the second observation information of each second agent.

[0195] In one possible implementation, the transceiver unit 1002 is specifically configured to: receive cooperative messages from at least one second agent.

[0196] Receive collaborative messages from at least one second agent forwarded by network devices.

[0197] In one possible implementation, the processing unit 1001 is further configured to: generate second information of the first agent and signature information of the first agent based on the first hidden state information;

[0198] The transceiver unit 1002 is also configured to: send a cooperation message of the first intelligent agent to the network device; the cooperation message of the first intelligent agent includes the second information of the first intelligent agent and the signature information of the first intelligent agent.

[0199] In one possible implementation, in determining the action to be performed based on the first hidden state information and the aggregation information, the processing unit 1001 is specifically used for:

[0200] The first hidden state information is used as the input to the policy network, and the aggregated information is used as the bias of the policy network. The action to be executed is then predicted by the policy network.

[0201] Alternatively, the first hidden state information and the aggregation information can be used as inputs to the policy network, which can then predict the actions to be performed.

[0202] It should be noted that the implementation of each unit described in FIG10 or FIG11 can also correspond to the description of the embodiments shown in FIG2 to FIG8. Furthermore, the beneficial effects of the multi-agent information processing device described in FIG10 or FIG11 can be described in the corresponding description of the embodiments shown in FIG2 to FIG8, and will not be repeated here.

[0203] Based on the description of the above method and device embodiments, this application also provides a terminal device. Please refer to FIG12, which is a schematic diagram of the structure of a terminal device provided in this application embodiment. The terminal device includes at least a processor 1201, a memory 1202, and a communication interface 1203, which are interconnected via a bus 1204. This terminal device can be used to execute relevant steps of a multi-agent information processing method. The terminal device can be an agent in a MARL with spontaneous communication or a chip within that agent, etc. The processor 1201 in the terminal device is used to read the computer program code stored in the memory 1202 and execute the method of any one of the embodiments shown in FIG2 to FIG8.

[0204] The memory 1202 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), and is used to store related computer programs and data.

[0205] The processor 1201 can be one or more central processing units (CPUs). When the processor 1201 is a CPU, the CPU can be a single-core CPU or a multi-core CPU.

[0206] For example, when the terminal device is a transmitting device, the processor 1201 in the terminal device can be used to read one or more programs stored in the memory 1202 and perform the following operations:

[0207] Acquire the first hidden state information of the first observation information; the first observation information is the local observation information of the first agent.

[0208] Acquire aggregated information; aggregated information is obtained by aggregating the first information of at least one second agent, and the first information of each second agent is obtained by encoding the second observation information; the second observation information is the local observation information of each second agent.

[0209] The action to be performed is determined based on the first hidden state information and the aggregate information.

[0210] It should be noted that the implementation of each operation can also correspond to the description of the method in any of the embodiments shown in Figures 2 to 8.

[0211] It should be noted that although the terminal device shown in Figure 12 only illustrates the processor 1201, memory 1202, communication interface 1203, and bus 1204, those skilled in the art should understand that in specific implementations, the terminal device may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the terminal device may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the terminal device may only include the devices necessary for implementing the embodiments of this application, and not necessarily all the devices shown in Figure 12.

[0212] This application also provides a chip, including a processor for calling and running a computer program from a memory, causing a device with the chip installed to perform the method described in any of the embodiments shown in Figures 2 to 8 above. This chip may be a chip in a terminal device.

[0213] This application also provides a computer-readable storage medium (memory) storing a computer program that, when executed, implements the method described in any of the embodiments shown in Figures 2 to 8. It is understood that the computer-readable storage medium here can include both built-in storage media within a device and extended storage media supported by the device. The computer-readable storage medium provides storage space containing the device's operating system. Furthermore, one or more computer programs suitable for loading and execution by the device's processor are also stored in this storage space. It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one computer-readable storage medium located remotely from the aforementioned processor.

[0214] This application also provides a computer program product, which includes computer program code. When the computer program code is run by a terminal device, the method flow described in any one of the embodiments in Figures 2 to 8 is implemented.

[0215] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0216] It should be understood that the processor mentioned in the embodiments of this application can be a CPU, or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0217] It should also be understood that the memory mentioned in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory can be ROM, Programmable Read-Only Memory (PROM), EPROM, Electrically Erasable Programmable Read-Only Memory (EEPROM), or flash memory. Volatile memory can be RAM, which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), Synchlink Dynamic Random Access Memory (SLDRAM), and Direct Rambus RAM (DR RAM).

[0218] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, the memory (storage module) is integrated into the processor.

[0219] It should be noted that the memories described herein are intended to include, but are not limited to, these and any other suitable types of memories.

[0220] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0221] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely exemplary. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0222] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0223] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0224] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. In the textual description of this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0225] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0226] The modules in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0227] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A multi-agent information processing method, characterized by, The method is applied to a first agent, and comprises: obtaining first hidden state information of first observation information, wherein the first observation information is local observation information of the first agent; obtaining aggregated information, wherein the aggregated information is obtained by aggregating first information of at least one second agent, the first information of each second agent is obtained by encoding second observation information, and the second observation information is local observation information of each second agent; determining an action to be performed based on the first hidden state information and the aggregated information.

2. The method of claim 1, wherein, The method further comprises: receiving a collaboration message from the at least one second agent, wherein the collaboration message of each second agent comprises the first information of each second agent; determining a weight of the first information of the at least one second agent based on the first hidden state information; aggregating the first information of the at least one second agent based on the weight of the first information of the at least one second agent to obtain the aggregated information.

3. The method of claim 2, wherein, The collaboration message of each second agent further comprises signature information of each second agent, and the signature information of each second agent is generated based on second hidden state information, wherein the second hidden state information is hidden state information of the second observation information of each second agent. The method further comprises: obtaining query information based on the first hidden state information; calculating the weight of the first information of each second agent by using the query information and the signature information of each second agent.

4. The method of claim 3, wherein, The weight of the first information of each of the second intelligent agents satisfies: wherein a i a vector representing a weight composition of the first information of each of the second agents, denotes the transpose of the query information, k1,..., k i-1 denotes the transpose of the query information, k1,..., k i+1 denotes the transpose of the query information, k1,..., k U denotes the signature information of the at least one second agent, U denotes the total number of agents.

5. The method of claim 1, wherein, Before the aggregated information is obtained, the method further comprises: generating the query information based on the first hidden state information; sending the query information to a network device. The method further comprises: receiving the aggregated information sent by the network device, wherein the aggregated information is obtained by the network device based on the query information and signature information of the at least one second agent, the signature information of each second agent is generated based on second hidden state information, and the second hidden state information is hidden state information of the second observation information of each second agent.

6. The method of claim 2, wherein, The method further comprises: receiving the collaboration message from the at least one second agent forwarded by the network device.

7. The method according to claim 5 or 6, characterized in that, After the first hidden state information of the first observation information is obtained, the method further comprises: generating second information of the first agent and signature information of the first agent based on the first hidden state information; sending a collaboration message of the first agent to the network device, wherein the collaboration message of the first agent comprises the second information of the first agent and the signature information of the first agent.

8. The method according to any one of claims 1 to 7, characterized in that, The method further comprises: The first hidden state information is taken as an input of a policy network, and the aggregated information is taken as a bias of the policy network, and an action to be performed is predicted by the policy network; Or the first hidden state information and the aggregated information are taken as inputs of the policy network, and the action to be performed is predicted by the policy network.

9. A multi-agent information processing device, characterized by comprising: The device is applied to a first agent, and the device comprises a processing unit; the processing unit is configured to: obtain first hidden state information of first observation information; the first observation information is local observation information of the first agent; obtain aggregated information; the aggregated information is obtained by aggregating first information of at least one second agent; the first information of each second agent is obtained by encoding second observation information; the second observation information is local observation information of each second agent; determine an action to be performed based on the first hidden state information and the aggregated information.

10. The apparatus of claim 9, wherein, The device further comprises a transceiver unit; in terms of obtaining the aggregated information, the processing unit is specifically configured to: receive a cooperation message from the at least one second agent through the transceiver unit; the cooperation message of each second agent comprises the first information of each second agent; determine a weight of the first information of the at least one second agent based on the first hidden state information; aggregate the first information of the at least one second agent based on the weight of the first information of the at least one second agent, to obtain the aggregated information.

11. The apparatus of claim 10, wherein, The cooperation message of each second agent further comprises signature information of each second agent; the signature information of each second agent is generated based on second hidden state information; the second hidden state information is hidden state information of the second observation information of each second agent; in terms of determining the weight of the first information of the at least one second agent based on the first hidden state information, the processing unit is specifically configured to: obtain query information; the query information is generated based on the first hidden state information; adopt the query information and the signature information of each second agent to calculate the weight of the first information of each second agent.

12. The apparatus of claim 11, wherein, a weight of the first information of each of the second intelligent agents satisfies: wherein a i a vector representing a weight composition of the first information of each of the second agents, denotes the transpose of the query information, k1,..., k i-1 i+1 U denotes the signature information of the at least one second agent, U denotes the total number of agents.​​ 13. The apparatus of claim 9, wherein, The device further comprises a transceiver unit; the processing unit is further configured to generate query information based on the first hidden state information; the transceiver unit is configured to send the query information to a network device; in terms of obtaining the aggregated information, the processing unit is specifically configured to: receive the aggregated information sent by the network device through the transceiver unit; the aggregated information is obtained by the network device based on the query information and the signature information of the at least one second agent, and the aggregation of the first information of the at least one second agent; the signature information of each second agent is generated based on second hidden state information; the second hidden state information is hidden state information of the second observation information of each second agent.

14. The apparatus of claim 10, wherein, in terms of receiving the cooperation message from the at least one second agent, the transceiver unit is specifically configured to: receive the cooperation message from the at least one second agent forwarded by the network device.

15. The apparatus of claim 13 or 14, wherein, The processing unit is further configured to generate second information of the first agent and signature information of the first agent based on the first hidden state information. The transceiving unit is further configured to send a cooperation message of the first agent to the network device, the cooperation message of the first agent including the second information of the first agent and the signature information of the first agent.

16. The apparatus of any one of claims 9-15, wherein, In determining the action to be performed based on the first hidden state information and the aggregated information, the processing unit is specifically configured to: input the first hidden state information into a policy network and input the aggregated information into a bias of the policy network, and predict the action to be performed through the policy network; or input the first hidden state information and the aggregated information into the policy network, and predict the action to be performed through the policy network.

17. A terminal device, comprising: A computer program product comprises a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code configured such that, on execution by a processor, the processor performs the method of any one of claims 1-8.

18. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program for execution by a device, the computer program configured such that, on execution, the device performs the method of any one of claims 1-8.

19. A computer program product, characterised in that, The computer program product, when executed by a terminal device, causes the terminal device to perform the method of any one of claims 1-8.

Images