Node, scheduler, method, and multi-agent intelligent network to provide distributed attention and arrange nodes

Abstract

A method for arranging nodes of a multi-agent intelligent network providing distributed attention into K layers of nodes, where, each node sending information on the node's capabilities to a scheduler, which receives a network model for a multi-agent intelligent network. The network model includes one or more layers of linear attention and the scheduler receiving the information on each node's capabilities. The scheduler emulating a topology of nodes based on communication and computation constraints. The scheduler assigning portions of the network model to the layers and the scheduler assigning each node to one of the layers. The scheduler sends information to each node on from which node that node should receive input information from and to which node that node should provide output information. Moreover, the scheduler also sends weights to each node is assigned to, and the weights include the weights of the layer that the node is assigned.

Description

[0001] NODE, SCHEDULER, METHOD, AND MULTI-AGENT INTELLIGENT NETWORK TO PROVIDE DISTRIBUTED ATTENTION AND ARRANGE NODES

[0002] TECHNICAL FIELD

[0003] The present disclosure relates generally to the field of wireless communication network management and, more specifically, to a node, a scheduler, and a method for arranging nodes of a multi-agent intelligent network providing distributed attention. Furthermore, the present disclosure relates more specifically to the multi-agent intelligent network for providing distributed attention and arranging the nodes, such as by neural network split and deployment over a network of devices.

[0004] BACKGROUND

[0005] In rapidly advancing domain of communication systems, upcoming future communication systems are anticipated to heavily rely on cooperation of distributed agents that are used to combine processing compatibility of each of the intelligent agents and produce results. Moreover, each of the distributed agents is used to jointly process multimodal data (e.g., images, videos, or sensing data collected by each of the distributed agents) to adapt to changing environments, solve various complex tasks, and share obtained data with other distributed agents, such as by distributing all relevant data (i.e., tokens) across the distributed agents.

[0006] Conventional communication systems, such as conventional self-attention mechanisms cannot be computed in a distributed manner and require all the relevant data (i.e., the tokens) to be transmitted to a central server for joint computation. On the other hand, the transmission of the relevant data from each of the distributed agents to the central server is not feasible due to limitations of communication bandwidth and privacy constraints. Moreover, certain attempts have been made to overcome the communication bandwidth and privacy constraints, such as by using the self-attention of a transformer, ring attention, retentive networks, message-passing neural networks, and the like. However, such attempts fail due to many reasons, such as unavailability of the relevant data in a single place, complex computational cost, restrictive data flow, and the like. Therefore, there exists a technical problem of how to split a large model into smaller segments for deployment across a distributed network having distributed agents in order to efficiently load large model segments.

[0007] Therefore, in light of the foregoing discussion, there exists a need to overcome the aforementioned drawbacks associated with the conventional nodes, conventional schedulers, and conventional methods for providing distributed attention and arranging the nodes.

[0008] SUMMARY

[0009] The present disclosure provides a node of a multi-agent intelligent network, a scheduler of the multi-agent intelligent network, and a method for the node of the multi-agent intelligent network for providing distributed attention and arranging the nodes. The present disclosure provides a solution to the existing problem of how to split a large model into smaller segments for deployment across a distributed network having distributed agents in order to efficiently load large model segments. An objective of the present disclosure is to provide a solution that overcomes at least partially the problems encountered in the prior art and provides the node of the multi-agent intelligent network, the scheduler of the multi-agent network, and the method for the node of the multi-agent intelligent network for providing distributed attention and arranging the nodes.

[0010] One or more objectives of the present disclosure are achieved by the solutions provided in the enclosed independent claims. Advantageous implementations of the present disclosure are further defined in the dependent claims. In one aspect, the present disclosure provides a method for arranging nodes of a multi-agent intelligent network providing distributed attention into K layers of nodes. The multi-agent intelligent network comprises a scheduler and the nodes. Each of the K layers comprises a plurality of nodes, and there is a first input layer of nodes, one or more intermediate layers of nodes, and a final layer of nodes. Moreover, each node is arranged to process input information through a multi-head linear attention model to provide output information, the nodes of the first layer are arranged to receive input tokens as input information, the nodes of the one or more intermediate layers are arranged to receive input information from one or more nodes of a previous layer and send output information to one or more nodes of a subsequent layer, and the nodes of the final layer are arranged to provide output information as output labels. The method comprises each node sending information on the node’s capabilities to the scheduler, the scheduler receiving a network model for the multi-agent intelligent network. The network model comprises one or more layers of linear attention, and the scheduler receives the information on each node’s capabilities. The scheduler emulating a topology of nodes based on communication and computation constraints, the scheduler assigning portions of the network model to the layers, the scheduler assigning each node to one of the layers, the scheduler sends information to each node from which node that node should receive input information from and to which node that node should provide output information to, and the scheduler also sends weights to each node is assigned to, wherein the weights include the weights of the layer that the node is assigned to.

[0011] The method is beneficial for arranging the nodes into K layers, which enables each node to handle complex tasks by distributing computation across multiple nodes, which further enhances parallel processing and scalability, as each node can focus on a specific portion of the overall task, improving performance and efficiency. Moreover, the method provides a structured flow of information from raw data to final output, streamlining data processing and ensuring that each layer handles the corresponding specific function, reducing confusion and improving processing speed. In addition, the multi-head linear attention models enable the nodes to focus on different parts of the input simultaneously, improving the network model’s ability to capture complex relationships in the data. This enhances the accuracy and depth of the model's predictions and outputs. Furthermore, by virtue of having the first layer directly process input tokens, the network model ensures that data is handled as soon as it enters the network model, minimizing latency and allowing the downstream layers to focus on higher-level processing. Furthermore, by virtue of sharing the capabilities (e.g., computing power, communication constraints), each node can enable the scheduler to make informed decisions on task allocation, optimizing resource usage and ensuring that tasks are assigned based on node strengths, leading to better overall performance. In addition, by virtue of receiving the network model, the scheduler gains the blueprint for how the network should function, including how attention layers are structured. Therefore, the method enables the scheduler to assign tasks and resources in alignment with the model’s architecture, ensuring that each node’s contribution fits into the overall model design. By virtue of knowing each node’s computation and communication capabilities allows the scheduler to optimize task distribution, ensuring that nodes are neither over- nor underutilized. Therefore, the method improves efficiency, minimizes delays, and ensures balanced workload distribution across the network. In addition, emulating the node topology allows the scheduler to simulate how the nodes will interact under real-world conditions, ensuring that communication bottlenecks or computational limitations are accounted for before assigning tasks, which leads to more efficient network operation and fewer performance issues. Moreover, by virtue of dividing the network model into portions and assigning them to layers, the scheduler ensures that each layer handles its designated portion of the task, allowing for modular processing and efficient multi-layered computation. In addition, assigning nodes to specific layers ensures that tasks are distributed logically across the network, which prevents duplication of work and ensures that each layer has the necessary resources (nodes) to carry out its assigned tasks, improving the overall efficiency of the network. Furthermore, by explicitly defining communication pathways between the nodes, the scheduler reduces confusion and ensures smooth data flow between layers, which avoids unnecessary delays and communication errors, improving the system’s reliability and speed. Finally, sending the correct weights to each node ensures that the network model operates as intended, with each node applying the appropriate transformations during processing, which preserves the integrity of the model's learning and inference processes, leading to accurate and consistent outputs across the network model. In another aspect, the present disclosure provides a multi-agent intelligent network configured to provide distributed attention into K layers of nodes and to arrange the nodes. The multi-agent intelligent network comprises a scheduler and the nodes, and the each of the K layers comprises a plurality of nodes. Moreover, there is a first input layer of nodes, one or more intermediate layers of nodes and a final layer of nodes. Furthermore, each node is arranged to process input information through a multihead linear attention model to provide output information, the nodes of the first layer are arranged to receive input tokens as input information, the nodes of the one or more intermediate layers are arranged to receive input information from one or more nodes of a previous layer and send output information to one or more nodes of a subsequent layer, and the nodes of the final layer are arranged to provide output information as output labels. Moreover, each node is configured to send information on the node’s capabilities to the scheduler, the scheduler is configured to receive a network model for the multi-agent intelligent network. The network model comprises one or more layers of linear attention, and the scheduler is configured to receive the information on each node’s capabilities. The scheduler is further configured to emulate a topology of nodes based on communication and computation constraints, assign portions of the network model to the layers, assign each node to one of the layers, send information to each node on from which node that node should receive input information from and to which node that node should provide output information to, and send weights to each node is assigned to, wherein the weights include the weights of the layer that the node is assigned to.

[0012] The multi-agent intelligent network achieves all the advantages and technical effects of the method of the present disclosure.

[0013] In yet another aspect, there is provided a node configured to operate in the multi-agent intelligent network configured to provide distributed attention into K layers of nodes and to arrange the nodes. The multi-agent intelligent network comprises a scheduler and the nodes, and the each of the K layers comprises a plurality of nodes. Moreover, there is a first input layer of nodes, one or more intermediate layers of nodes and a final layer of nodes. Furthermore, each node is arranged to process input information through a multi-head linear attention model to provide output information, the nodes of the first layer are arranged to receive input tokens as input information, the nodes of the one or more intermediate layers are arranged to receive input information from one or more nodes of a previous layer and send output information to one or more nodes of a subsequent layer, and the nodes of the final layer are arranged to provide output information as output labels. The node is configured to send information on the node’s capabilities to the scheduler.

[0014] The node achieves all the advantages and technical effects of the multi-agent intelligent network of the present disclosure.

[0015] In yet another implementation, there is provided a scheduler configured to operate in a multi-agent intelligent network configured to provide distributed attention into K layers of nodes and to arrange the nodes. The multi-agent intelligent network comprises a scheduler and the nodes, and the each of the K layers comprises a plurality of nodes. Moreover, there is a first input layer of nodes, one or more intermediate layers of nodes and a final layer of nodes. Furthermore, each node is arranged to process input information through a multi-head linear attention model to provide output information, the nodes of the first layer are arranged to receive input tokens as input information, the nodes of the one or more intermediate layers are arranged to receive input information from one or more nodes of a previous layer and send output information to one or more nodes of a subsequent layer, and the nodes of the final layer are arranged to provide output information as output labels. Moreover, each node is configured to send information on the node’s capabilities to the scheduler. The scheduler is configured to receive a network model for the multi-agent intelligent network. The network model comprises one or more layers of linear attention and receive the information on each node’s capabilities. The scheduler is further configured to emulate a topology of nodes based on communication and computation constraints, assign portions of the network model to the layers, assign each node to one of the layers, send information to each node on from which node that node should receive input information from and to which node that node should provide output information to, and send weights to each node is assigned to, wherein the weights include the weights of the layer that the node is assigned to. The scheduler achieves all the advantages and technical effects of the multi-agent intelligent network of the present disclosure.

[0016] It is to be appreciated that all the aforementioned implementation forms can be combined.

[0017] It has to be noted that all devices, elements, circuitry, units, and means described in the present application could be implemented in the software or hardware elements or any kind of combination thereof. All steps which are performed by the various entities described in the present application, as well as the functionalities described to be performed by the various entities are intended to mean that the respective entity is adapted to or configured to perform the respective steps and functionalities. Even if, in the following description of specific embodiments, a specific functionality or step to be performed by external entities is not reflected in the description of a specific detailed element of that entity that performs that specific step or functionality, it should be clear for a skilled person that these methods and functionalities can be implemented in respective software or hardware elements or any kind of combination thereof. It will be appreciated that features of the present disclosure are susceptible to being combined in various combinations without departing from the scope of the present disclosure as defined by the appended claims.

[0018] Additional aspects, advantages, features, and objects of the present disclosure would be made apparent from the drawings and the detailed description of the illustrative implementations construed in conjunction with the appended claims that follow.

[0019] BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The summary above, as well as the following detailed description of illustrative embodiments, is better understood when read in conjunction with the appended drawings. For the purpose of illustrating the present disclosure, exemplary constructions of the disclosure are shown in the drawings. However, the present disclosure is not limited to specific methods and instrumentalities disclosed herein. Moreover, those in the art will understand that the drawings are not to scale. Wherever possible, like elements have been indicated by identical numbers.

[0021] Embodiments of the present disclosure will now be described, by way of example only, with reference to the following diagrams wherein:

[0022] FIG. 1 is a block diagram that illustrates a multi-agent intelligent network providing distributed attention, in accordance with an embodiment of the present disclosure;

[0023] FIG. 2 is a flowchart of a method for arranging nodes of a multi-agent intelligent network, in accordance with an embodiment of the present disclosure;

[0024] FIG. 3A and 3B are scenario-based diagrams that illustrate sharing topology information with a server, in accordance with an embodiment of the present disclosure;

[0025] FIG. 3C is a scenario-based diagram that illustrates various operations for splitting a network model at a server, in accordance with an embodiment of the present disclosure;

[0026] FIG. 3D and 3E are scenario-based diagrams that illustrate deploying a network model on distributed nodes, in accordance with an embodiment of the present disclosure;

[0027] FIG. 4 is a diagram that illustrates an exemplary scenario of a vehicle monitored by a network of cameras, in accordance with an embodiment of the present disclosure;

[0028] FIG. 5 is a block diagram that depicts a multi-agent intelligent network, in accordance with an embodiment of the present disclosure; and

[0029] FIG. 6 is a block diagram that depicts a multi-agent intelligent network, in accordance with an embodiment of the present disclosure.

[0030] In the accompanying drawings, an underlined number is employed to represent an item over which the underlined number is positioned or an item to which the underlined number is adjacent. A non-underlined number relates to an item identified by a line linking the non-underlined number to the item. When a number is non-underlined and accompanied by an associated arrow, the non-underlined number is used to identify a general item at which the arrow is pointing.

[0031] DETAILED DESCRIPTION OF EMBODIMENTS

[0032] The following detailed description illustrates embodiments of the present disclosure and ways in which they can be implemented. Although some modes of carrying out the present disclosure have been disclosed, those skilled in the art would recognize that other embodiments for carrying out or practicing the present disclosure are also possible.

[0033] FIG. 1 is a block diagram that illustrates a multi-agent intelligent network providing distributed attention, in accordance with an embodiment of the present disclosure. With reference to FIG. 1, there is shown a multi-agent intelligent network 100 providing distributed attention. The multi-agent intelligent network 100 includes a network model 104 and a server 108. The network model 104 (or a multi-head network model) includes a plurality of model layers, such as a first layer 114A, a second layer 114B, kth layer 114K, nth layer 114N, and n+l-layer 114N+1. Furthermore, the multi-agent intelligent network 100 includes system layers, such as a first system layer 116A and a second system layer 116B that further includes one or more nodes, such as a first node 118A, a second node 118B, a third node 118C, and a fourth node 118D.

[0034] There is provided the multi-agent intelligent network 100 providing distributed attention. The multi-agent intelligent network 100 provides distributed attention by allowing multiple nodes to independently process and focus on different parts of the data while communicating and collaborating with each other to provide an enhanced efficiency, robustness, and data privacy while handling complex and large-scale operations in a decentralized and efficient manner with reduced communication overhead. The multi-agent intelligent network 100 includes the scheduler 110 and the nodes, such as a first node 118A, a second node 118B, a third node 118C, and a fourth node 118D. Moreover, each of the K layers includes a plurality of nodes, such as, there is a first input layer of nodes, one or more intermediate layers of nodes, and a final layer of nodes.

[0035] The network model 104 (or the multi-head network model) is configured to receive a model input data 102 (i.e., a sequence of input tokens), such as a query translated into tokens and provides a model output data 106, which is a sequence of output tokens translated into a response that is further utilized for computing the distributed attention. Moreover, the server 108 includes the scheduler 110, which is configured to control, schedule, or train the network model 104 based on the information (e.g., the information received at operation 112), such as information on system capabilities, number or nodes, and the like. The network model 104 is deployed to compute distributed attention within the multi-agent intelligent network 100 and includes multiple model layers, such as the first layer 114A, the second layer 114B, the kth layer 114k, the nth layer 114N, and the nth+l-layer 114N+1. Moreover, each layer of the model layers includes multiple heads and is configured to transmit the set of local inputs along with the memory states (i.e., partially processed tokens) to the other layer of the multi-head network model 104, such as at operation 120. For example, at operation 120A, the nth layer 114N is configured to receive the local input embeddings from the previous node and, further, at operation 120B, transmits the local input embeddings (or the partially processed tokens) to the nth+l-layer 114N+1.

[0036] Furthermore, the nodes of the first layer are arranged to receive input tokens as input information, the nodes of the one or more intermediate layers are arranged to receive input information from one or more nodes of a previous layer, and send output information to one or more nodes of a subsequent layer. In accordance with an embodiment, the input information includes one or more tokens to be processed, and output information of a previous node, if any. There is further provided that each node, such as the first node 118A, the second node 118B, the third node 118C, and the fourth node 118D is arranged to process the input information through the multi-head linear attention model to provide the output information. In an implementation, the multi-head linear attention model includes multiple layers of a multi-head linear attention, which is used to process the one or more tokens of the input information. In accordance with an embodiment, the output information of a node includes memory states indicating information about processed input information and a counter, which indicates the total number of processed tokens by the node. In an example, the memory states store information about the input that has been processed, and the counter keeps track of how many tokens have passed through the node. By virtue of retaining the processed information, the node can maintain context and continuity between the neighbor nodes, which is required for tasks that rely on sequential data processing. Moreover, by virtue of tracking the total number of processed tokens, the node can manage and control the process flow, ensuring each node accurately distinguishes the volume of data processed by the node, which may be desirable for performance, debugging, or decision-making tasks.

[0037] Furthermore, each model layer of the multi-head network model 104 includes at least one system layer, such as the first system layer 116A and the second system layer 116B, that are assigned to one or more nodes. In other words, each of the nodes executes one or more model layers, and one or more heads, such as a first head 124A, a second head 124B, a third head 124C, and a fourth head 124D, are assigned to the one or more nodes. Moreover, each of the node executes all heads of the executed model layers. For example, the first head 124A, the second head 124B, the third head 124C, and the fourth head 124D are assigned to the second node 118B. As a result, the multi-agent intelligent network 100 is configured to provide distributed attention by allowing each node within the multi-agent intelligent network 100 to determine the combined node memory state and further transmit the same to the other nodes along with the set of local memory states.

[0038] In operation, each node (e.g., the first node 118A) is configured to send information on the node’s capabilities to the scheduler 110 of the server 108. In accordance with an embodiment, the information on the node’s capabilities include an identifier of the node, and computation constraints including computing resources of the node. The information on the node’s capabilities further includes communication constraints including an identifier (ID) of at least one neighbouring node and throughput to at least one of the neighbouring devices. In an implementation, each node of the first system layer 116A sends the information as a message. In an example, the message includes an ID of the sending node, which enables efficient task distribution and targeted communication between nodes, improving overall system coordination and performance. In such an example, the message further includes the computing resources of the node. In accordance with an embodiment, the computing resources further includes memory resources. Therefore, the message includes the memory resources of the node, which enables efficient task distribution and targeted communication between the neighbouring nodes, improving overall coordination and performance. In such an example, the message further includes an ID of at least one neighbouring node, and available communication throughputs between each node. This ensures that no node is overloaded and that tasks are distributed optimally, leading to balanced workload s and preventing system bottleneck, which increases efficiency by ensuring that nodes operate within corresponding capacity, reducing the likelihood of performance degradation or failures due to overloading.

[0039] In accordance with an embodiment, each node is configured to receive a set of local input embeddings (Xn, X1-X3) as the input information. For example, the first node 118A is configured to receive the set of local input embeddings (Xn, X1-X3) as the input information. Thereafter, the first node 118A is configured to apply positional embedding to the set of local input embeddings (Xn, X1-X3). In this embodiment, applying the positional embeddings to the set of local input embeddings (Xn, X1-X3) enables the first node 108A to incorporate positional context, improving the ability of the network model 104 to understand the order and relationships within the input information for more accurate processing and predictions. Furthermore, the first node is configured to determine a set of keys (Kn) by multiplying the set of local input embeddings (Xn, X1-X3) with a learned key matrix (WK) and also determine a set of values (Vn) by multiplying the set of local input embeddings (Xn, XI- X3) with a learned value matrix (WV). In such embodiment, by virtue of determining the set of keys (Kn) and the set of values (Vn) through the learned key matrix (WK) and the learned value matrix (WV), the first node 108A enables efficient attention mechanisms. This allows the first node 108A to selectively focus on relevant parts of the set of local input embeddings (Xn, X1-X3), enhancing the ability of the network model 104 to capture meaningful relationships and improve overall processing accuracy in various tasks, such as language understanding or sequence modelling. Thereafter, the first node 108A if configured to determine a set of queries (Qn) by multiplying the set of local input embeddings (Xn, X1-X3) with a learned query matrix (WQ). By virtue of determining the set of queries (Qn) through the learned query matrix (WQ), the first node 108A facilitates the attention mechanism by enabling comparison between the set of queries (Qn), the set of keys (Kn), and the set of values (Vn). This allows the network model 104 to focus on relevant parts of the set of local input embeddings (Xn, X1-X3), improving the accuracy and efficiency of context-aware data processing.

[0040] The first node 108A is further configured to determine a set of local memory states (S’n) based on the set of local input embeddings (Xn, X1-X3), a learned key matrix (WK), and a learned value matrix (WV). The set of local memory states (S’n) retain important contextual information from the of local input embeddings (Xn, X1-X3), the learned key matrix (WK), and the and a learned value matrix (WV), which enabled and improved long-term data retention and relevance in processing. This enhances the efficiency of attention mechanisms, allowing the first node 108A to focus on critical data, which further improves accuracy, adaptability, and computational efficiency in handling complex tasks. Moreover, the first node 108A if further configured to receive one or more node memory states (Si) from at least one of the one or more other nodes (L, M), the received one or more node memory states forming a set of node memory states (Sn). The technical benefit is that by receiving the one or more node memory states (Si) from one or more other nodes (L, M) and forming the set of node memory states (Sn), the ability of the first node (108A) is improved to aggregate and integrate distributed information from the one or more other nodes (L, M). This interaction builds upon the previously determined the set of local memory states (S’n) by incorporating external contextual data, allowing the network model 104 to capture broader relationships and dependencies across nodes, which improves overall system coordination, scalability, and processing accuracy in distributed tasks.

[0041] The first node 108A is further configured to determine a set of combined node memory states (Sn) based on the received set of node memory states (Sn) and the set of local memory states (S’n). In such embodiment, the set of combined node memory states (Sn) are beneficial to capture global patterns and relationships across distributed nodes, improving the overall accuracy, scalability, and coordination of the network model 104. The set of combined node memory states (Sn) further ensure a comprehensive view of both local and network-wide data for more informed decision-making. The first node 108A is further configured to provide the set of combined node memory states (Sn) to at least one next node, and determine an output (On) based on the determined set of queries (Qn) and the set of combined node memory states (Sn). As a result, the first node 108A facilitates seamless data flow and interaction between the one or more other nodes (L, M). This ensures that the output reflects both local and distributed information, improving the ability of the network model 104 to generate accurate and contextually aware results while enhancing scalability and coordination across the network.

[0042] The scheduler 110 is further configured to receive the network model 104 for the multi-agent intelligent network 100, such as the network model 104 includes one or more layers of linear attention. Thereafter, the scheduler 110 is configured to receive the information on each node’s capabilities. After that, the scheduler 110 is configured to emulate a topology of nodes based on communication and computation constraints. The scheduler 110 of the server 108 further uses information received from the plurality of nodes to emulate a virtual network topology together with its constraints. As a result, the scheduler 110 can evaluate how many tokens a selected node can process at a single time, and how many tokens a node can send to or receive from other nodes (e.g., from the neighbouring nodes). The scheduler 110 is further configured to use the received information to emulate the topology of the nodes together with corresponding communication and computation constraints. After that, the scheduler 110 is configured to split the network model 104 into separate layers (all heads are kept together) and assign the layers to the nodes such that each node receives weights of one or more consecutive attention layers.

[0043] The scheduler 110 is further configured to assign portions of the network model to the layers, and also to assign each node to one of the layers. As a result, the scheduler 110 is configured to efficiently assign portions of the network model 104 to specific layers, optimizing task division across, which further improves computational resource management, ensuring that each layer handles a well-defined part of the model, reducing redundancy and boosting processing speed. Moreover, by virtue of assigning each node to a specific layer, the scheduler 110 ensures structured parallel processing, where each node focuses on its dedicated layer's computation, which is beneficial to improve load balancing, reducing contention between nodes and enhancing overall throughput of the network model 104.

[0044] In accordance with an embodiment, the portions are non-overlapping and a portion includes at least one full multi-head linear attention layer of the network model. The non-overlapping portions ensure that each layer is processed independently, avoiding duplicate computations. By including a full multi-head linear attention layer, each portion maximizes the utilization of attention mechanisms (e.g., multi-head attention), which improves the capability of the network model 104 to capture complex relationships across input data.

[0045] In accordance with an embodiment, the scheduler 110 is configured to assign the portions of the network model to the layers such that all heads of each layer of the network model are kept together. By virtue of keeping all heads of each layer of the network model 104 together ensures cohesive and consistent attention processing, which reduces inter-layer communication overhead and maintains the integrity of the attention mechanism, resulting in improved efficient performance of the network model 104.

[0046] In an implementation, the scheduler 110 is configured to appropriately split the network model 104 into separate layers, assign layers to each device, and orchestrate processing and communication between the devices. In an example, an M41node may receive weights of a layer k, of layers k and k+1, but not of layers k and k+2. In accordance with an embodiment, weights for at least one node further include weights of one or more subsequent layers of the network model. Moreover, the same layer could be assigned to multiple devices. In addition, the scheduler 110 informs each node from whom the node should receive information (i.e., states or processed tokens) and to whom to send processed data (i.e., states or processed tokens).

[0047] Thereafter, the scheduler 110 is configured to send information to each node on from which node that node should receive input information from and to which node that node should provide output information to, and send weights to each node is assigned to, such as the weights include the weights of the layer that the node is assigned to . In accordance with an embodiment, the scheduler 110 is configured to send to each node: an identifier of the scheduler 110, an identifier of the portions that the scheduler 110 is sending, the weights of the layers of the network model 104, an identifier of devices from whom the first node 118A should receive input information, and an identifier of devices to whom the first node 118A should send output information.

[0048] In an implementation, the scheduler 110 is configured to send a message to the nodes, such as the message includes an ID of the server 108 as well as an ID of the layers that the server 108 is sending. In an example, the message includes an ID of the scheduler 110. In an example, the message includes an ID of the server 108 as well as an ID of the layers that the server 108. In an example, the message includes an ID of the scheduler 110. In such example, the message further includes weights of the selected attention layers (e.g., all heads are kept together). In addition, the message further incudes an ID of nodes from whom the first node 118A can receive memory states as well as an ID of the nodes from whom the first node 118A can receive tokens output by a preceding layer. In such example, the message further includes an ID of the nodes to whom the first node 118A can send locally computed memory states as well as an ID of the nodes to whom the first node 118A should send locally processed output tokens.

[0049] In an implementation, the network model 104 consists of K layers of linear attention, and the scheduler 110 groups node into l<k<=K adjoining clusters of approximately similar combined computing power and throughput. To each cluster, the scheduler 110 assigns one or more consecutive layers of the network model 104. Thereafter, the scheduler 110 orders nodes in each cluster such that, node (k,l) is directly connected to node (k,2), node (k, 2) is connected to node (k, 3), and so on. Moreover, node (1, 1) is directly connected to node (2,1), node (2,1) is connected to node (3,1), and so on. As a result, the multi-agent intelligent network 100 providing distributed attention into K layers of nodes, can be used in various useful applications in domains, such as autonomous driving, distributed intelligent computing, smart manufacturing, e-health care, and the lie. An exemplary implementation of such multi-agent intelligent network with a scenario of traffic monitoring is further illustrated and explained in FIG. 4, in which a network of intelligent cameras follows and analyse the movement of a car in a city. When the car moves forward along the streets, it sequentially enters and leaves the field of view of multiple cameras. Each time the car leaves the field of view of a particular camera, the corresponding camera processes the recorded video and computes a memory state, which summarizes all relevant information about the car. The computed state is then forwarded to the next camera, which can process its own recorded video while attending to information acquired by previous cameras.

[0050] The multi-agent intelligent network 100 is beneficial for arranging the nodes into K layers, which enables each node to handle complex tasks by distributing computation across multiple nodes, which further enhances parallel processing and scalability, as each node can focus on a specific portion of the overall task, improving performance and efficiency. Moreover, the multiagent intelligent network 100 provides a structured flow of information from raw data to final output, streamlining data processing and ensuring that each layer handles the corresponding specific function, reducing confusion and improving processing speed. In addition, the multi-head linear attention models enable each node to focus on different parts of the input simultaneously, improving the ability of the network model 104 to capture complex relationships in the input information, which enhances the accuracy and depth of the predictions and outputs of the network model 104. Furthermore, by virtue of having the first layer 114A directly process input tokens, the network model 104 ensures that data is handled as soon as it enters the network model 104, minimizing latency and allowing the downstream layers to focus on higher-level processing. Furthermore, by virtue of sharing the capabilities (e.g., computing power, communication constraints), each node can enable the scheduler 110 to make informed decisions on task allocation, optimizing resource usage and ensuring that tasks are assigned based on node strengths, leading to better overall performance. In addition, by virtue of receiving the network model 104, the scheduler 110 gains the blueprint for how the network model 104 should function, including how attention layers are structured. Therefore, the multi-agent intelligent network 100 enables the scheduler 110 to assign tasks and resources in alignment with the architecture of the network model 104, ensuring that each node’s contribution fits into the overall model design. By virtue of knowing each node’s computation and communication capabilities allows the scheduler 110 to optimize task distribution, ensuring that nodes are neither over- nor underutilized. Therefore, the method improves efficiency, minimizes delays, and ensures balanced workload distribution across the network model 104. In addition, emulating the node topology allows the scheduler 110 to simulate how the nodes will interact under real- world conditions, ensuring that communication bottlenecks or computational limitations are accounted for before assigning tasks, which leads to more efficient network operation and fewer performance issues. Moreover, by virtue of dividing the network model 104 into portions and assigning the portions to the layers, the scheduler 110 ensures that each layer handles a designated portion of the task, allowing for modular processing and efficient multi-layered computation. In addition, assigning nodes to specific layers ensures that tasks are distributed logically across the network, which prevents duplication of work and ensures that each layer has the necessary resources (nodes) to carry out its assigned tasks, improving the overall efficiency of the network model 104. Furthermore, by explicitly defining communication pathways between the nodes, the scheduler 110 reduces confusion and ensures smooth data flow between layers, which avoids unnecessary delays and communication errors, improving the reliability and speed of the multi-agent intelligent network 100. Finally, sending the correct weights to each node ensures that the network model 104 operates as intended, with each node applying the appropriate transformations during processing, which preserves the integrity of the model's learning and inference processes, leading to accurate and consistent outputs across the network model 104.

[0051] FIG. 2 is a flowchart of a method for arranging nodes of a multi-agent intelligent network, in accordance with an embodiment of the present disclosure. FIG. 2 is described in conjunction with elements from FIG. 1. With reference to FIG. 2, there is provided a flowchart of a method 200 method for arranging nodes of the multi-agent intelligent network 100 providing distributed attention into K layers of nodes. The method 200 includes steps 202-to-222. The multi-agent intelligent network 100 includes the scheduler 110 and the nodes, such as a first node 118A, a second node 118B, a third node 118C, and a fourth node 118D. Moreover, each of the K layers includes a plurality of nodes, such as, there is a first input layer of nodes, one or more intermediate layers of nodes, and a final layer of nodes. In addition, each node is arranged to process input information through a multi-head linear attention model to provide output information. The method 200 comprises, receiving, by the nodes of the first layer, input tokens as input information, the nodes of the one or more intermediate layers are arranged to receive input information from one or more nodes of a previous layer, and send output information to one or more nodes of a subsequent layer. In accordance with an embodiment, the input information includes one or more tokens to be processed, and output information of a previous node, if any. In an implementation, the server 108 includes large model, such as the multi-head linear attention model that includes multiple layers of multi-head linear attention, which is used to process the one or more tokens of the input information. There is further provided that each node, such as the first node 118A, the second node 118B, the third node 118C, and the fourth node 118D is arranged to process the input information through the multi-head linear attention model to provide the output information. In accordance with an embodiment, the output information of a node includes memory states indicating information about processed input information and a counter, which indicates the total number of processed tokens by the node. In an example, the memory states store information about the input that has been processed, and the counter keeps track of how many tokens have passed through the node. By virtue of retaining the processed information, the node can maintain context and continuity between the neighbor nodes, which is required for tasks that rely on sequential data processing. Moreover, by virtue of tracking the total number of processed tokens, the node can manage and control the process flow, ensuring each node accurately distinguishes the volume of data processed by the node, which may be desirable for performance, debugging, or decision-making tasks.

[0052] Furthermore, each of the model layers of the multi-head network model 104 includes at least one system layer, such as the first system layer 116A and the second system layer 116B, that are assigned to one or more nodes. In other words, each node executes one or more model layers, and one or more heads, such as a first head 124A, a second head 124B, a third head 124C, and a fourth head 124D, are assigned to the one or more nodes. Moreover, each node executes all heads of the executed model layers. For example, the first head 124A, the second head 124B, the third head 124C, and the fourth head 124D are assigned to the second node 118B. As a result, the multi-agent intelligent network 100 is configured to provide distributed attention by allowing each node within the multi-agent intelligent network 100 to determine the combined node memory state and further transmit the same to the other nodes along with the set of local memory states.

[0053] At step 202, the method 200 comprises, sending, by each node (e.g.,the first node 118A), information on the node’s capabilities to the scheduler 110. In accordance with an embodiment, the information on the node’s capabilities include an identifier of the node, computation constraints including computing resources of the node. The information on the node’s capabilities further includes communication constraints including an identifier (ID) of at least one neighbouring node and throughput to at least one of the neighbouring devices. In an implementation, each node of the first system layer 116A sends the information as a message. In an example, the message includes an ID of the sending node, which enables efficient task distribution and targeted communication between nodes, improving overall system coordination and performance. In such an example, the message further includes the computing resources of the node. In accordance with an embodiment, the computing resources further includes memory resources. Therefore, the message includes the memory resources of the node, which enables efficient task distribution and targeted communication between the neighbouring nodes, improving overall coordination and performance. In such an example, the message further includes the ID of at least one neighbouring node, and available communication throughputs between each node. This ensures that no node is overloaded and that tasks are distributed optimally, leading to balanced workload s and preventing system bottleneck, which increases efficiency by ensuring that nodes operate within corresponding capacity, reducing the likelihood of performance degradation or failures due to overloading.

[0054] In accordance with an embodiment, each node is configured to receive a set of local input embeddings (Xn, X1-X3) as the input information. For example, the first node 118A is configured to receive the set of local input embeddings (Xn, X1-X3) as the input information. Thereafter, the first node 118A is configured to apply positional embedding to the set of local input embeddings (Xn, X1-X3). In this embodiment, applying the positional embeddings to the set of local input embeddings (Xn, X1-X3) enables the first node 108A to incorporate positional context, improving the ability of the network model 104 to understand the order and relationships within the input information for more accurate processing and predictions. Furthermore, the first node is configured to determine a set of keys (Kn) by multiplying the set of local input embeddings (Xn, X1-X3) with a learned key matrix (WK) and also determine a set of values (Vn) by multiplying the set of local input embeddings (Xn, XI- X3) with a learned value matrix (WV). In such embodiment, by virtue of determining the set of keys (Kn) and the set of values (Vn) through the learned key matrix (WK) and the learned value matrix (WV), the first node 108A enables efficient attention mechanisms. This allows the first node 108A to selectively focus on relevant parts of the set of local input embeddings (Xn, X1-X3), enhancing the ability of the network model 104 to capture meaningful relationships and improve overall processing accuracy in various tasks, such as language understanding or sequence modelling. Thereafter, the first node 108A if configured to determine a set of queries (Qn) by multiplying the set of local input embeddings (Xn, X1-X3) with a learned query matrix (WQ). By virtue of determining the set of queries (Qn) through the learned query matrix (WQ), the first node 108A facilitates the attention mechanism by enabling comparison between the set of queries (Qn), the set of keys (Kn), and the set of values (Vn). This allows the network model 104 to focus on relevant parts of the set of local input embeddings (Xn, X1-X3), improving the accuracy and efficiency of context-aware data processing.

[0055] The first node 108A is further configured to determine a set of local memory states (S’n) based on the set of local input embeddings (Xn, X1-X3), a learned key matrix (WK), and a learned value matrix (WV). The set of local memory states (S’n) retain important contextual information from the of local input embeddings (Xn, X1-X3), the learned key matrix (WK), and the and a learned value matrix (WV), which enabled and improved long-term data retention and relevance in processing. This enhances the efficiency of attention mechanisms, allowing the first node 108A to focus on critical data, which further improves accuracy, adaptability, and computational efficiency in handling complex tasks. Moreover, the first node 108A if further configured to receive one or more node memory states (Si) from at least one of the one or more other nodes (L, M), the received one or more node memory states forming a set of node memory states (Sn). The technical benefit is that by receiving the one or more node memory states (Si) from one or more other nodes (L, M) and forming the set of node memory states (Sn), the ability of the first node (108A) is improved to aggregate and integrate distributed information from the one or more other nodes (L, M). This interaction builds upon the previously determined the set of local memory states (S’n) by incorporating external contextual data, allowing the network model 104 to capture broader relationships and dependencies across nodes, which improves overall system coordination, scalability, and processing accuracy in distributed tasks.

[0056] The first node 108A is further configured to determine a set of combined node memory states (Sn) based on the received set of node memory states (Sn) and the set of local memory states (S’n). In such embodiment, the set of combined node memory states (Sn) are beneficial to capture global patterns and relationships across distributed nodes, improving the overall accuracy, scalability, and coordination of the network model 104. The set of combined node memory states (Sn) further ensure a comprehensive view of both local and network-wide data for more informed decision-making. The first node 108A is further configured to provide the set of combined node memory states (Sn) to at least one next node, and determine an output (On) based on the determined set of queries (Qn) and the set of combined node memory states (Sn). As a result, the first node 108A facilitates seamless data flow and interaction between the one or more other nodes (L, M). This ensures that the output reflects both local and distributed information, improving the ability of the network model 104 to generate accurate and contextually aware results while enhancing scalability and coordination across the network.

[0057] At step 204, the method 200 comprises, receiving, by the scheduler 110, the network model 104 for the multi-agent intelligent network 100, such as the network model 104 includes one or more layers of linear attention. At step 206, the method 200 comprises, receiving, by the scheduler 110, the information on each node’s capabilities. After that, the scheduler 110 is configured to emulate a topology of nodes based on communication and computation constraints. The scheduler 110 of the server 108 further uses information received from the plurality of nodes to emulate a virtual network topology together with its constraints. As a result, the scheduler 110 can evaluate how many tokens a selected node can process at a single time, and how many tokens a node can send to or receive from other nodes (e.g., from neighbouring nodes). The scheduler 110 is further configured to use the received information to emulate the topology of the nodes together with corresponding communication and computation constraints. After that, the scheduler 110 is configured to split the network model 104 into separate layers (all heads are kept together) and assign the layers to the nodes such that each node receives weights of one or more consecutive attention layers.

[0058] At step 208, the method 200 comprises, assigning, by scheduler 110, portions of the network model to the layers. As a result, the scheduler 110 is configured to efficiently assign portions of the network model 104 to specific layers, optimizing task division across, which further improves computational resource management, ensuring that each layer handles a well-defined part of the model, reducing redundancy and boosting processing speed. At step 210, the method 200 comprises, assigning, by scheduler 110, assigning each node to one of the layers. By virtue of assigning each node to a specific layer, the scheduler 110 ensures structured parallel processing, where each node focuses on its dedicated layer's computation, which is beneficial to improve load balancing, reducing contention between nodes and enhancing overall model throughput.

[0059] In accordance with an embodiment, the portions are non-overlapping and a portion includes at least one full multi-head linear attention layer of the network model. The non-overlapping portions ensure that each layer is processed independently, avoiding duplicate computations. By including a full multi-head linear attention layer, each portion maximizes the utilization of attention mechanisms (e.g., multi-head attention), which improves the capability of the network model 104 to capture complex relationships across input data. In accordance with an embodiment, the scheduler 110 assigns the portions of the network model to the layers such that all heads of each layer of the network model 104 are kept together. By virtue of keeping all heads of each layer of the network model 104 together ensures cohesive and consistent attention processing, which reduces inter-layer communication overhead and maintains the integrity of the attention mechanism, resulting in improved efficient performance of the network model 104.

[0060] In an implementation, the scheduler 110 is configured to appropriately split the network model 104 into separate layers, assign layers to each device, and orchestrate processing and communication between the devices. In an example, an M41node may receive weights of a layer k, of layers k and k+1, but not of layers k and k+2. In accordance with an embodiment, weights for at least one node further include weights of one or more subsequent layers of the network model. Moreover, the same layer could be assigned to multiple devices. In addition, the scheduler 110 informs each node from whom the node should receive information (i.e., states or processed tokens) and to whom to send processed data (i.e., states or processed tokens).

[0061] Thereafter, the scheduler 110 is configured to send information to each node on from which node that node should receive input information from and to which node that node should provide output information to, and send weights to each node is assigned to, such as the weights include the weights of the layer that the node is assigned to. In accordance with an embodiment, the scheduler 110 sends to each node: an identifier of the scheduler, an identifier of the portions that the scheduler is sending, the weights of the layers of the network model, an identifier of devices from whom the node should receive input information, and an identifier of devices to whom the node should send output information.

[0062] In an implementation, the scheduler 110 is configured to send a message to the nodes, such as the message includes an ID of the server 108 as well as an ID of the layers that the server 108 is sending. In an example, the message includes an ID of the scheduler 110. In an example, the message includes an ID of the server 108 as well as an ID of the layers that the server 108. In an example, the message includes an ID of the scheduler 110. In such example, the message further includes weights of the selected attention layers (e.g., all heads are kept together). In addition, the message further incudes an ID of nodes from whom the first node 118A can receive memory states as well as an ID of the nodes from whom the first node 118A can receive tokens output by a preceding layer. In such example, the message further includes an ID of the nodes to whom the first node 118A can send locally computed memory states as well as an ID of the nodes to whom the first node 118A should send locally processed output tokens.

[0063] In an implementation, the network model 104 consists of K layers of linear attention, and the scheduler 110 groups nodes into l<k<=K adjoining clusters of approximately similar combined computing power and throughput. To each cluster, the scheduler 110 assigns one or more consecutive layers of the network model 104. Thereafter, the scheduler 110 orders nodes in each cluster such that, node (k,l) is directly connected to node (k,2), node (k, 2) is connected to node (k, 3), and so on. Moreover, node (1 , 1 ) is directly connected to node (2,1 ), node (2,1 ) is connected to node (3,1 ), and so on. As a result, the proposed multiagent intelligent network 100 providing distributed attention into K layers of nodes, can be used in various useful applications in domains, such as autonomous driving, distributed intelligent computing, smart manufacturing, e-health care, and the he. An exemplary implementation of such multi-agent intelligent network with a scenario of traffic monitoring is further illustrated and explained in FIG. 4, in which a network of intelligent cameras follows and analyse the movement of a car in a city. When the car moves forward along the streets, it sequentially enters and leaves the field of view of multiple cameras. Each time the car leaves the field of view of a particular camera, the corresponding camera processes the recorded video and computes a memory state, which summarizes all relevant information about the car. The computed state is then forwarded to the next camera, which can process its own recorded video while attending to information acquired by previous cameras.

[0064] The method 200 achieves all the advantages and technical effects of the multi-agent intelligent network 100 of the present disclosure.

[0065] The steps 202 to 214 are only illustrative, and other alternatives can also be provided where one or more steps are added, one or more steps are removed, or one or more steps are provided in a different sequence without departing from the scope of the claims herein.

[0066] There is further provided a computer program product comprising program instructions for performing the method 200 when executed by one or more processors in the multi-agent intelligent network 100. The computer program product is implemented as an algorithm, embedded in a software stored in a non-transitory computer-readable storage medium. The non-transitory computer-readable storage means may include, but are not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. Examples of implementation of computer-readable storage medium, but are not limited to, Electrically Erasable Programmable Read-Only Memory (EEPROM), Random Access Memory (RAM), Read Only Memory (ROM), Hard Disk Drive (HDD), Flash memory, a Secure Digital (SD) card, Solid-State Drive (SSD), a computer-readable storage medium, and / or CPU cache memory.

[0067] FIG. 3A and 3B are scenario-based diagrams that illustrates sharing a topology information with a server, in accordance with an embodiment of the present disclosure. FIGs. 3A and 3B are described in conjunction with elements from FIG. 1 and FIG. 2. With reference to FIG. 3A, there is shown the first node 118A in communication with the server 108. Similarly, with reference to FIG. 3B, there is shown that each of the first node 118A, the second node 118B, the third node 118C, the fourth node 118D, and a fifth node 118E are in communication with the server 108.

[0068] In an implementation, the first node 118A is configured to receive input tokens as input information from one or more nodes of a previous layer. In accordance with an embodiment, the input information includes one or more tokens to be processed, and output information of a previous node. Thereafter, the first node 118 A is configured to process the input information through the multi-head linear attention model, such as to provide the output information. In an implementation, the multi-head linear attention model includes multiple layers of a multi-head linear attention, which is used to process the one or more tokens of the input information. In accordance with an embodiment, the output information of the first node 118A includes memory states indicating information about processed input information and a counter, which indicates the total number of processed tokens by the node. In an example, the memory states store information about the input that has been processed, and the counter keeps track of how many tokens have passed through the node. By virtue of retaining the processed information, the first node 118A can maintain context and continuity between the neighbor nodes, which is required for tasks that rely on sequential data processing. Moreover, by virtue of tracking the total number of processed tokens, the first node 118A can manage and control the process flow, ensuring the first node 118A accurately distinguishes the volume of data processed by the first node 118A, which may be desirable for performance, debugging, or decision-making tasks. After that, the first node 118A is further configured to send information on the node’s capabilities to the server 108, such as through operation 302, as shown in FIG. 3A. Similarly, each of the second node 118B, the third node 118C, the fourth node 118D, and the fifth node 118E are also configured to send the information of the node’s capabilities to the server 108, as shown in FIG. 3B. In accordance with an embodiment, the information on the node’s capabilities includes an identifier of the first node 118A, computation constraints including computing resources of the first node 118A. The information on the node’s capabilities further includes communication constraints including an identifier (ID) of at least one neighbouring node and throughput to at least one of the neighbouring devices. In an implementation, the first node 118A sends the information as a message. In an example, the message includes ID of the first node 118A, which enables efficient task distribution and targeted communication between one or more nodes, improving overall coordination and performance. In such an example, the message further includes computing resources of the first node 118A. In accordance with an embodiment, the computing resources further includes memory resources. Therefore, the message includes the memory resources of the first node 118A, which enables efficient task distribution and targeted communication between the neighbouring nodes, improving overall coordination and performance. In such an example, the message further includes ID of at least one neighbouring node, and available communication throughputs between each node. This ensures that no node is overloaded and that tasks are distributed optimally, leading to balanced workload s and preventing system bottleneck, which increases efficiency by ensuring that nodes operate within corresponding capacity, reducing the likelihood of performance degradation or failures due to overloading.

[0069] FIG. 3C is a scenario-based diagram that illustrates various operations for splitting a network model at a server, in accordance with an embodiment of the present disclosure. FIG. 3C is described in conjunction with elements from FIG. 1, FIG. 2, FIG. 3A, and FIG. 3B. With reference to FIG. 3C, there is shown the server 108.

[0070] At operation 304A, the server 108 is configured to use the ID of each of the first node 118A, the second node 118B, the third node 118C, the fourth node 118D, and the fifth node 118E to reconstruct a virtual topology of the nodes. Thereafter, the server 108 is configured to use a computational power and memory capabilities of node to estimate the maximum number of tokens that each node can process at the same time. Furthermore, the server 108 is configured to use the information about throughput between the nodes to estimate the maximum number of tokens or states that an individual node can send to other nodes.

[0071] In an implementation, the network model 104 consists of K layers of linear attention and the server 108 groups node into l<k<=K adjoining clusters of approximately similar combined computing power and throughput, such as shown at operation 304B. To each cluster, the server 108 assigns one or more consecutive layers of the network model 104. For example, the first node 118A, the fourth node 118D, and the seventh node 108G are assigned to a first cluster 306A. Similarly, the second node 118B, the fifth node 118E, and the eight node 118H are assigned to a second cluster 306B. In addition, the third node 118C, the sixth node 118F, and a ninth node 1181 are assigned to a third cluster 306C. Thereafter, at operation 304C, the server 108 orders nodes in each cluster such that, node (k,l) is directly connected to node (k,2), node (k, 2) is connected to node (k, 3), and so on. Moreover, node (1, 1) is directly connected to node (2,1), node (2,1) is connected to node (3,1), and so on.

[0072] FIG. 3D and 3E are scenario-based diagrams that illustrates deploying a network model on distributed nodes, in accordance with an embodiment of the present disclosure. FIGs. 3D and 3E are described in conjunction with elements from FIG. 1 , FIG. 2, FIG. 3A, 3B, FIG. 3C. With reference to FIG. 3D, there is shown the first node 118A in communication with the server 108. Similarly, with reference to FIG. 3BE, there is shown that each of the first node 118A, the second node 118B, the third node 118C, the fourth node 118D, and the fifth node 118E are in communication with the server 108.

[0073] In an implementation, the server 108 is configured split the network model 104 into K layers and sends a message to each of the first node 118A, the second node 118B, the third node 118C, the fourth node 118D, and the fifth node 118E. In an example, the message includes an ID of the server 108 as well as an ID of the layers that the server 108. In an example, the message includes an ID of the scheduler 110. In such example, the message further includes weights of the selected attention layers (e.g., all heads are kept together). In addition, the message further incudes an ID of nodes from whom the first node 118A can receive memory states as well as an ID of the nodes from whom the first node 118A can receive tokens output by a preceding layer. In such example, the message further includes an ID of the nodes to whom the first node 118A can send locally computed memory states as well as an ID of the nodes to whom the first node 118A should send locally processed output tokens.

[0074] FIG. 4 is a diagram that illustrates an exemplary scenario of a vehicle monitored by a network of cameras, in accordance with an embodiment of the present disclosure. FIG. 4 is described in conjunction with elements from FIG. 1, FIG. 2, and FIG. 3A to FIG. 3E. With reference to FIG. 4, there is shown a diagram 400 that depicts a vehicle 402, which is moving in direction 404 and is monitored by the multi-agent intelligent network 100 of the cameras, such as a first camera 406A, a second camera 406B, and a third camera 406C.

[0075] In an implementation, the cameras, such as the first camera 406A, the second camera 406B, and the third camera 406C, are configured to provide the data, such as the captured frames (e.g., a first frame 408A, a second frame 408B, and a third frame 408C) to the associated nodes. For example, the first camera 406A is configured to provide the data captured to the first node 118A, and the second camera 406B is configured to provide the data captured to the second node 118B. Similarly, the third camera 406C is configured to provide the data captured to the third node 118C. Furthermore, each of the nodes transfers a last frame along with the combined memory state and the counter. For example, at operation 410, the first node 118 A is configured to transfer the combined memory state along with the counter and a last frame to the second node 118B, and at operation 412, the second node 118B is configured to transfer the last frame along with the combined memory state along with the counter. As a result, the multi-agent intelligent network 100 can be utilized for distributed linear attention, such as autonomous driving, distributed intelligent computing, smart manufacturing, e-health care, and the like.

[0076] FIG. 5 is a block diagram that depicts a multi-agent intelligent network, in accordance with an embodiment of the present disclosure. FIG. 5 is described in conjunction with elements from FIG. 1, FIG. 2, FIG. 3A to FIG. 3E, and FIG. 4. With reference to FIG. 5, there is shown the multi-agent intelligent network 100 providing distributed attention. The multi-agent intelligent network 100 includes the network model 104 and the server 108. The network model 104 (or a multi-head network model) includes a plurality of model layers, such as the first layer 114A, the second layer 114B, the kth layer 114K, the nth layer 114N, and the n+l-layer 114N+1. Furthermore, the multi-agent intelligent network 100 includes system layers, such as the first system layer 116A and the second system layer 116B that further includes one or more nodes, such as the first node 118A, the second node 118B, the third node 118C, and the fourth node 118D. There is further shown that the first node 118A includes a first memory 502, and a first network interface 504, and a first controller 506.

[0077] The first memory 502 is configured to store the computer executable instructions configured to be executed by the first node 118A. Examples of implementation of the first memory 502 may include but are not limited to, an Electrically Erasable Programmable Read-Only Memory (EEPROM), Dynamic Random-Access Memory (DRAM), Random Access Memory (RAM), Read-Only Memory (ROM), Hard Disk Drive (HDD), Flash memory, a Secure Digital (SD) card, Solid-State Drive (SSD), and / or CPU cache memory. The first network interface 504 refers is configured to allow the communication between the first node 118A and one or more other nodes, such as the second node 118B. Examples of the first network interface 504 may include but are not limited to a computer port, a network socket, a network interface controller (NIC), and any other network interface device. Similarly examples of the first controller 506 may include but are not limited to a central data processing device, a microprocessor, a microcontroller, a complex instruction set computing (CISC) processor, an application-specific integrated circuit (ASIC) processor, a reduced instruction set (RISC) processor, a very long instruction word (VLIW) processor, a state machine, and other processors or control circuitry.

[0078] In one aspect, a node, such as the first node 118A is configured to operate in the multi-agent intelligent network 100 configured to provide distributed attention into K layers of nodes and to arrange the nodes. The multi-agent intelligent network 100 includes the scheduler 110 and the nodes, such as a first node 118A, a second node 118B, a third node 118C, and a fourth node 118D. Moreover, each of the K layers includes a plurality of nodes, such as, there is a first input layer of nodes, one or more intermediate layers of nodes and a final layer of nodes.

[0079] In an implementation, each node, such as the first node 118A is arranged to process input information through a multi-head linear attention model to provide output information. Furthermore, the nodes of the first layer are arranged to receive input tokens as input information, the nodes of the one or more intermediate layers are arranged to receive input information from one or more nodes of a previous layer, and send output information to one or more nodes of a subsequent layer. In accordance with an embodiment, the input information includes one or more tokens to be processed, and output information of a previous node, if any. There is further provided that each node, such as the first node 118A, the second node 118B, the third node 118C, and the fourth node 118D is arranged to process the input information through the multi-head linear attention model to provide the output information. In an implementation, the multi-head linear attention model includes multiple layers of a multi-head linear attention, which is used to process the one or more tokens of the input information. In accordance with an embodiment, the output information of a node includes memory states indicating information about processed input information and a counter, which indicates the total number of processed tokens by the node. In an example, the memory states store information about the input that has been processed, and the counter keeps track of how many tokens have passed through the node. By virtue of retaining the processed information, the node can maintain context and continuity between the neighbor nodes, which is required for tasks that rely on sequential data processing. Moreover, by virtue of tracking the total number of processed tokens, the node can manage and control the process flow, ensuring each node accurately distinguishes the volume of data processed by the node, which may be desirable for performance, debugging, or decision-making tasks.

[0080] Furthermore, each model layer of the multi-head network model 104 includes at least one system layer, such as the first system layer 116A and the second system layer 116B, that are assigned to one or more nodes. In other words, each of the node executes one or more model layers and one or more heads, such as a first head 124A, a second head 124B, a third head 124C, and a fourth head 124D, are assigned to the one or more nodes. Moreover, each of the node executes all heads of the executed model layers. For example, the first head 124A, the second head 124B, the third head 124C, and the fourth head 124D are assigned to the second node 118B. As a result, the multi-agent intelligent network 100 is configured to provide distributed attention by allowing each node within the multi-agent intelligent network 100 to determine the combined node memory state and further transmit the same to the other nodes along with the set of local memory states.

[0081] The first node 118A is configured to send information on the node’s capabilities to the scheduler 110 of the server 108. In accordance with an embodiment, the information on the node’s capabilities includes an identifier of the node, computation constraints including computing resources of the node. The information on the node’s capabilities further includes communication constraints including an identifier (ID) of at least one neighbouring node and throughput to at least one of the neighbouring devices. In an implementation, each node of the first system layer 116A sends the information as a message. In an example, the message includes ID of the sending node, which enables efficient task distribution and targeted communication between nodes, improving overall system coordination and performance. In such an example, the message further includes computing resources of the node. In accordance with an embodiment, the computing resources further includes memory resources. Therefore, the message includes the memory resources of the node, which enables efficient task distribution and targeted communication between the neighbouring nodes, improving overall coordination and performance. In such an example, the message further includes ID of at least one neighbouring node, and available communication throughputs between each node. This ensures that no node is overloaded and that tasks are distributed optimally, leading to balanced workload s and preventing system bottleneck, which increases efficiency by ensuring that nodes operate within corresponding capacity, reducing the likelihood of performance degradation or failures due to overloading.

[0082] The first node 118A achieves all the advantages and technical effects of the multi-agent intelligent network 100 of the present disclosure.

[0083] FIG. 6 is a block diagram that depicts a multi-agent intelligent network, in accordance with an embodiment of the present disclosure. FIG. 6 is described in conjunction with elements from FIG. 1, FIG. 2, FIG. 3A to FIG. 3E, FIG. 4, and FIG. 5. With reference to FIG. 6, there is shown the multi-agent intelligent network 100 providing distributed attention. The multi-agent intelligent network 100 includes the network model 104 and the server 108. The network model 104 (or a multi-head network model) includes a plurality of model layers, such as the first layer 114A, the second layer 114B, the kth layer 114K, the nth layer 114N, and the n+l-layer 114N+1. Furthermore, the multi-agent intelligent network 100 includes system layers, such as the first system layer 116A and the second system layer 116B that further includes one or more nodes, such as the first node 118A, the second node 118B, the third node 118C, and the fourth node 118D. There is further shown that the server 108 includes the scheduler 110, which further includes a second memory 602, and a second network interface 604, and a second controller 606.

[0084] Examples of implementation of the second memory 602 may include but are not limited to, an Electrically Erasable Programmable Read-Only Memory (EEPROM), Dynamic Random-Access Memory (DRAM), Random Access Memory (RAM), Read-Only Memory (ROM), Hard Disk Drive (HDD), Flash memory, a Secure Digital (SD) card, Solid-State Drive (SSD), and / or CPU cache memory. Examples of the second network interface 604 may include but are not limited to a computer port, a network socket, a network interface controller (NIC), and any other network interface device. Similarly examples of the second controller 606 may include but are not limited to a central data processing device, a microprocessor, a microcontroller, a complex instruction set computing (CISC) processor, an application-specific integrated circuit (ASIC) processor, a reduced instruction set (RISC) processor, a very long instruction word (VLIW) processor, a state machine, and other processors or control circuitry.

[0085] Furthermore, the nodes of the first layer are arranged to receive input tokens as input information, the nodes of the one or more intermediate layers are arranged to receive input information from one or more nodes of a previous layer, and send output information to one or more nodes of a subsequent layer. In accordance with an embodiment, the input information includes one or more tokens to be processed, and output information of a previous node, if any. There is further provided that each node, such as the first node 118A, the second node 118B, the third node 118C, and the fourth node 118D is arranged to process the input information through the multi-head linear attention model to provide the output information. In an implementation, the multi-head linear attention model includes multiple layers of a multi-head linear attention, which is used to process the one or more tokens of the input information. In accordance with an embodiment, the output information of a node includes memory states indicating information about processed input information and a counter, which indicates the total number of processed tokens by the node. In an example, the memory states store information about the input that has been processed, and the counter keeps track of how many tokens have passed through the node. By virtue of retaining the processed information, the node can maintain context and continuity between the neighbor nodes, which is required for tasks that rely on sequential data processing. Moreover, by virtue of tracking the total number of processed tokens, the node can manage and control the process flow, ensuring each node accurately distinguishes the volume of data processed by the node, which may be desirable for performance, debugging, or decision-making tasks.

[0086] Furthermore, each model layer of the multi-head network model 104 includes at least one system layer, such as the first system layer 116A and the second system layer 116B, that are assigned to one or more nodes. In other words, each of the node executes one or more model layers and one or more heads, such as a first head 124A, a second head 124B, a third head 124C, and a fourth head 124D, are assigned to the one or more nodes. Moreover, each of the node executes all heads of the executed model layers. For example, the first head 124A, the second head 124B, the third head 124C, and the fourth head 124D are assigned to the second node 118B. As a result, the multi-agent intelligent network 100 is configured to provide distributed attention by allowing each node within the multi-agent intelligent network 100 to determine the combined node memory state and further transmit the same to the other nodes along with the set of local memory states.

[0087] In operation, each node (e.g., the first node 118A) is configured to send information on the node’s capabilities to the scheduler 110 of the server 108. In accordance with an embodiment, the information on the node’s capabilities includes an identifier of the node, computation constraints including computing resources of the node. The information on the node’s capabilities further includes communication constraints including an identifier (ID) of at least one neighbouring node and throughput to at least one of the neighbouring devices. In an implementation, each node of the first system layer 116A sends the information as a message. In an example, the message includes ID of the sending node, which enables efficient task distribution and targeted communication between nodes, improving overall system coordination and performance. In such an example, the message further includes computing resources of the node. In accordance with an embodiment, the computing resources further includes memory resources. Therefore, the message includes the memory resources of the node, which enables efficient task distribution and targeted communication between the neighbouring nodes, improving overall coordination and performance. In such an example, the message further includes ID of at least one neighbouring node, and available communication throughputs between each node. This ensures that no node is overloaded and that tasks are distributed optimally, leading to balanced workload s and preventing system bottleneck, which increases efficiency by ensuring that nodes operate within corresponding capacity, reducing the likelihood of performance degradation or failures due to overloading.

[0088] In accordance with an embodiment, each node is configured to receive a set of local input embeddings (Xn, X1-X3) as the input information. For example, the first node 118A is configured to receive the set of local input embeddings (Xn, X1-X3) as the input information. Thereafter, the first node 118A is configured to apply positional embedding to the set of local input embeddings (Xn, X1-X3). In this embodiment, applying the positional embeddings to the set of local input embeddings (Xn, X1-X3) enables the first node 108A to incorporate positional context, improving the ability of the network model 104 to understand the order and relationships within the input information for more accurate processing and predictions. Furthermore, the first node is configured to determine a set of keys (Kn) by multiplying the set of local input embeddings (Xn, X1-X3) with a learned key matrix (WK) and also determine a set of values (Vn) by multiplying the set of local input embeddings (Xn, XI- X3) with a learned value matrix (WV). In such embodiment, by virtue of determining the set of keys (Kn) and the set of values (Vn) through the learned key matrix (WK) and the learned value matrix (WV), the first node 108A enables efficient attention mechanisms. This allows the first node 108A to selectively focus on relevant parts of the set of local input embeddings (Xn, X1-X3), enhancing the ability of the network model 104 to capture meaningful relationships and improve overall processing accuracy in various tasks, such as language understanding or sequence modelling. Thereafter, the first node 108A if configured to determine a set of queries (Qn) by multiplying the set of local input embeddings (Xn, X1-X3) with a learned query matrix (WQ). By virtue of determining the set of queries (Qn) through the learned query matrix (WQ), the first node 108A facilitates the attention mechanism by enabling comparison between the set of queries (Qn), the set of keys (Kn), and the set of values ( Vn). This allows the network model 104 to focus on relevant parts of the set of local input embeddings (Xn, X 1 -X3 ), improving the accuracy and efficiency of context-aware data processing. The first node 108A is further configured to determine a set of local memory states (S’n) based on the set of local input embeddings (Xn, X1-X3), a learned key matrix (WK), and a learned value matrix (WV). The set of local memory states (S’n) retain important contextual information from the of local input embeddings (Xn, X1-X3), the learned key matrix (WK), and the and a learned value matrix (WV), which enabled and improved long-term data retention and relevance in processing. This enhances the efficiency of attention mechanisms, allowing the first node 108A to focus on critical data, which further improves accuracy, adaptability, and computational efficiency in handling complex tasks. Moreover, the first node 108A if further configured to receive one or more node memory states (Si) from at least one of the one or more other nodes (L, M), the received one or more node memory states forming a set of node memory states (Sn). The technical benefit is that by receiving the one or more node memory states (Si) from one or more other nodes (L, M) and forming the set of node memory states (Sn), the ability of the first node (108A) is improved to aggregate and integrate distributed information from the one or more other nodes (L, M). This interaction builds upon the previously determined the set of local memory states (S’n) by incorporating external contextual data, allowing the network model 104 to capture broader relationships and dependencies across nodes, which improves overall system coordination, scalability, and processing accuracy in distributed tasks.

[0089] The first node 108A is further configured to determine a set of combined node memory states (Sn) based on the received set of node memory states (Sn) and the set of local memory states (S’n). In such embodiment, the set of combined node memory states (Sn) are beneficial to capture global patterns and relationships across distributed nodes, improving the overall accuracy, scalability, and coordination of the network model 104. The set of combined node memory states (Sn) further ensure a comprehensive view of both local and network-wide data for more informed decision-making. The first node 108A is further configured to provide the set of combined node memory states (Sn) to at least one next node, and determine an output (On) based on the determined set of queries (Qn) and the set of combined node memory states (Sn). As a result, the first node 108A facilitates seamless data flow and interaction between the one or more other nodes (L, M). This ensures that the output reflects both local and distributed information, improving the ability of the network model 104 to generate accurate and contextually aware results while enhancing scalability and coordination across the network.

[0090] The scheduler 110 is further configured to receive the network model 104 for the multi-agent intelligent network 100, such as the network model 104 includes one or more layers of linear attention. Thereafter, the scheduler 110 is configured to receive the information on each node’s capabilities. After that, the scheduler 110 is configured to emulate a topology of nodes based on communication and computation constraints. The scheduler 110 of the server 108 further uses information received from the plurality of nodes to emulate a virtual network topology together with its constraints. As a result, the scheduler 110 can evaluate how many tokens a selected node can process at a single time, and how many tokens a node can send to or receive from other nodes (e.g., from neighbouring nodes). The scheduler 110 is further configured to use the received information to emulate the topology of the nodes together with corresponding communication and computation constraints. After that, the scheduler 110 is configured to split the network model 104 into separate layers (all heads are kept together) and assigns the layers to the nodes such that each node receives weights of one or more consecutive attention layers.

[0091] The scheduler 110 is further configured to assign portions of the network model to the layers, and also to assign each node to one of the layers. As a result, the scheduler 110 is configured to efficiently assigns portions of the network model 104 to specific layers, optimizing task division across, which further improves computational resource management, ensuring that each layer handles a well-defined part of the model, reducing redundancy and boosting processing speed. Moreover, by virtue of assigning each node to a specific layer, the scheduler 110 ensures structured parallel processing, where each node focuses on its dedicated layer's computation, which is beneficial to improve load balancing, reducing contention between nodes and enhancing overall model throughput.

[0092] In accordance with an embodiment, the portions are non-overlapping and a portion includes at least one full multi-head linear attention layer of the network model. The non-overlapping portions ensure that each layer is processed independently, avoiding duplicate computations. By including a full multi-head linear attention layer, each portion maximizes the utilization of attention mechanisms (e.g., multi-head attention), which improves the capability of the network model 104 to capture complex relationships across input data.

[0093] In accordance with an embodiment, the scheduler assigning the portions of the network model to the layers such that all heads of each layer of the network model are kept together. By virtue of keeping all heads of each layer of the network model 104 together ensures cohesive and consistent attention processing, which reduces inter-layer communication overhead and maintains the integrity of the attention mechanism, resulting in improve efficient performance of the network model 104.

[0094] In an implementation, the scheduler 110 is configured to appropriately splits the network model 104 into separate layers, assigns layers to each device, and orchestrates processing and communication between the devices. In an example, an M41node may receive weights of a layer k, of layers k and k+1 , but not of layers k and k+2. In accordance with an embodiment, weights for at least one node further includes weights of one or more subsequent layers of the network model. Moreover, the same layer could be assigned to multiple devices. In addition, the scheduler 110 informs each node from whom the node should receive information (i.e., states or processed tokens) and to whom send processed data (i.e., states or processed tokens).

[0095] Thereafter, the scheduler 110 is configured to send information to each node on from which node that node should receive input information from and to which node that node should provide output information to, and send weights to each node is assigned to, such as the weights include the weights of the layer that the node is assigned to . In accordance with an embodiment, the scheduler 110 sending to each node: an identifier of the scheduler, an identifier of the portions that the scheduler is sending, the weights of the layers of the network model, an identifier of devices from whom the node should receive input information, and an identifier of devices to whom the node should send output information.

[0096] In an implementation, the scheduler 110 is configured to send a message to the nodes, such as the message includes an ID of the server 108 as well as an ID of the layers that the server 108 is sending. In an example, the message includes an ID of the scheduler 110. In an example, the message includes an ID of the server 108 as well as an ID of the layers that the server 108. In an example, the message includes an ID of the scheduler 110. In such example, the message further includes weights of the selected attention layers (e.g., all heads are kept together). In addition, the message further incudes an ID of nodes from whom the first node 118A can receive memory states as well as an ID of the nodes from whom the first node 118A can receive tokens output by a preceding layer. In such example, the message further includes an ID of the nodes to whom the first node 118A can send locally computed memory states as well as an ID of the nodes to whom the first node 118A should send locally processed output tokens.

[0097] In an implementation, the network model 104 consists of K layers of linear attention and the scheduler 110 groups node into l<k<=K adjoining clusters of approximately similar combined computing power and throughput. To each cluster, the scheduler 110 assigns one or more consecutive layers of the network model 104. Thereafter, the scheduler 110 orders nodes in each cluster such that, node (k,l) is directly connected to node (k,2), node (k, 2) is connected to node (k, 3), and so on. Moreover, node (1 , 1 ) is directly connected to node (2,1 ), node (2,1 ) is connected to node (3,1 ), and so on. As a result, the proposed multiagent intelligent network 100 providing distributed attention into K layers of nodes, can be used in various useful applications in domains, such as, autonomous driving, distributed intelligent computing, smart manufacturing, e-health care, and the lie.

[0098] The scheduler 110 achieves all the advantages and technical effects of the multi-agent intelligent network 100 of the present disclosure.

[0099] Modifications to embodiments of the present disclosure described in the foregoing are possible without departing from the scope of the present disclosure as defined by the accompanying claims. Expressions such as "including", "comprising", "incorporating", "have", "is" used to describe and claim the present disclosure are intended to be construed in a non-exclusive manner, namely allowing for items, components or elements not explicitly described also to be present. Reference to the singular is also to be construed to relate to the plural. The word "exemplary" is used herein to mean "serving as an example, instance or illustration". Any embodiment described as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments or to exclude the incorporation of features from other embodiments. The word "optionally" is used herein to mean "is provided in some embodiments and not provided in other embodiments". It is appreciated that certain features of the present disclosure, which are, for clarity, described in the context of separate embodiments, may also be provided in combination in a single embodiment. Conversely, various features of the invention, which are, for brevity, described in the context of a single embodiment, may also be provided separately or in any suitable combination or as suitable in any other described embodiment of the disclosure.

Claims

CLAIMS1. A method (200) for arranging nodes of a multi-agent intelligent network (100) providing distributed attention into K layers of nodes, wherein the multi-agent intelligent network (100) comprises a scheduler and the nodes, and wherein the each of the K layers comprises a plurality of nodes, and wherein there is a first input layer of nodes, one or more intermediate layers of nodes and a final layer of nodes, wherein each node is arranged to process input information through a multi-head linear attention model to provide output information, the nodes of the first layer are arranged to receive input tokens as input information, the nodes of the one or more intermediate layers are arranged to receive input information from one or more nodes of a previous layer and send output information to one or more nodes of a subsequent layer, and the nodes of the final layer are arranged to provide output information as output labels, wherein the method (200) comprises each node sending information on the node’s capabilities to the scheduler (110), the scheduler receiving a network model for the multi-agent intelligent network (100), wherein the network model comprises one or more layers of linear attention, and the scheduler (110) receiving the information on each node’s capabilities, whereby the scheduler (110) emulating a topology of nodes based on communication and computation constraints, the scheduler (110) assigning portions of the network model to the layers, the scheduler (110) assigning each node to one of the layers, the scheduler (110) sends information to each node on from which node that node should receive input information from and to which node that node should provide output information to, and the scheduler (110) also sends weights to each node is assigned to, wherein the weights include the weights of the layer that the node is assigned to.

2. The method (200) according to claim 1, wherein the input information comprises one or more tokens to be processed, and output information of a previous node, if any.

3. The method (200) according to any preceding claim, wherein the output information of a node comprises memory states indicating information about processed input information and a counter which indicates the total number of processed tokens by the node.

4. The method (200) according to any receding claim, wherein the portions are non-overlapping and a portion includes at least one full multi-head linear attention layer of the network model.

5. The method (200) according to claim 4, wherein the method (200) further comprises the scheduler assigning the portions of the network model to the layers such that all heads of each layer of the network model are kept together.

6. The method (200) according to any preceding claim, wherein weights for at least one node further includes weights of one or more subsequent layers of the network model.

7. The method (200) according to any preceding claim, wherein the information on the node’s capabilities includes an identifier of the node, computation constraints including computing resources of the node, and communication constraints including an identifier of at least one neighbouring node, and22throughput to at least one of the neighbouring devices.

8. The method (200) according to claim 6, wherein the computing resources further includes memory resources.

9. The method (200) according to any preceding claim, wherein the method (200) further comprises the scheduler (110) sending to each node: an identifier of the scheduler, an identifier of the portions that the scheduler is sending, the weights of the layers of the network model, an identifier of devices from whom the node should receive input information, an identifier of devices to whom the node should send output information.

10. The method (200) according to any preceding claim, wherein the method (200) further comprises each node receiving a set of local input embeddings (Xn, X1-X3) as the input information, applying positional embedding to the set of local input embeddings (Xn, X1-X3), determining a set of keys (Kn) by multiplying the set of local input embeddings (Xn, X1-X3) with a learned key matrix (WK), determining a set of values (Vn) by multiplying the set of local input embeddings (Xn, X1-X3) with a learned value matrix (Wv), determining a set of queries (Qn) by multiplying the set of local input embeddings (Xn, X1-X3) with a learned query matrix (WQ), wherein the method (200) further comprises the node determining a set of local memory states (S’n) based on the set of local input embeddings (Xn, X1-X3), a learned key matrix (WK) and a learned value matrix (Wv), receiving one or more node memory states (S1) from at least one of the one or more other nodes (L, M), the received one or more node memory states forming a set of node memory states (Sn), determining a set of combined node memory states (Sn) based on the received set of node memory states (Sn) and the set of local memory states (S’n), providing the set of combined node memory states (Sn) to at least one next node, and determine an output (On) based on the determined set of queries (Qn) and the set of combined node memory states (Sn).

11. A computer program product comprising program instructions for performing the method (200) according to claim 10, when executed by one or more processors in a multi-agent intelligent network.

12. A multi-agent intelligent network (100) configured to provide distributed attention into K layers of nodes and to arrange the nodes, wherein the multi-agent intelligent network (100) comprises a scheduler and the nodes, and wherein the each of the K layers comprises a plurality of nodes, and wherein there is a first input layer of nodes, one or more intermediate layers of nodes and a final layer of nodes, wherein each node is arranged to process input information through a multi-head linear attention model to provide output information, the nodes of the first layer are arranged to receive input tokens as input information, the nodes of the one or more intermediate layers are arranged to receive input information from one or more nodes of a previous layer and send output information to one or more nodes of a subsequent layer, and the nodes of the final layer are arranged to provide output information as output labels, wherein each node is configured to send information on the node’s capabilities to the scheduler, the scheduler is configured to receive a network model for the multi-agent intelligent network, wherein the network model comprises one or more layers of linear attention, andthe scheduler is configured to receive the information on each node’s capabilities, whereby the scheduler is configured to emulate a topology of nodes based on communication and computation constraints, assign portions of the network model to the layers, assign each node to one of the layers, send information to each node on from which node that node should receive input information from and to which node that node should provide output information to, and send weights to each node is assigned to, wherein the weights include the weights of the layer that the node is assigned to.

13. A node configured to operate in a multi-agent intelligent network (100) configured to provide distributed attention into K layers of nodes and to arrange the nodes, wherein the multi-agent intelligent network (100) comprises a scheduler and the nodes, and wherein the each of the K layers comprises a plurality of nodes, and wherein there is a first input layer of nodes, one or more intermediate layers of nodes and a final layer of nodes, wherein each node is arranged to process input information through a multi-head linear attention model to provide output information, the nodes of the first layer are arranged to receive input tokens as input information, the nodes of the one or more intermediate layers are arranged to receive input information from one or more nodes of a previous layer and send output information to one or more nodes of a subsequent layer, and the nodes of the final layer are arranged to provide output information as output labels, wherein the node is configured to send information on the node’s capabilities to the scheduler.

14. A scheduler configured to operate in a multi-agent intelligent network (100) configured to provide distributed attention into K layers of nodes and to arrange the nodes, wherein the multi-agent intelligent network (100) comprises a scheduler (110) and the nodes, and wherein the each of the K layers comprises a plurality of nodes, and wherein there is a first input layer of nodes, one or more intermediate layers of nodes and a final layer of nodes, wherein each node is arranged to process input information through a multi-head linear attention model to provide output information, the nodes of the first layer are arranged to receive input tokens as input information, the nodes of the one or more intermediate layers are arranged to receive input information from one or more nodes of a previous layer and send output information to one or more nodes of a subsequent layer, and the nodes of the final layer are arranged to provide output information as output labels, wherein each node is configured to send information on the node’s capabilities to the scheduler (110) , wherein the scheduler (110) is configured to receive a network model for the multi-agent intelligent network, wherein the network model comprises one or more layers of linear attention, and receive the information on each node’s capabilities, whereby the scheduler (110) is configured to emulate a topology of nodes based on communication and computation constraints, assign portions of the network model to the layers, assign each node to one of the layers, send information to each node on from which node that node should receive input information from and to which node that node should provide output information to, and send weights to each node is assigned to, wherein the weights include the weights of the layer that the node is assigned to.

Images