Node, method, and multi-agent intelligent network for providing distributed attention

Abstract

A node of a multi-agent intelligent network providing distributed attention configured to receive a set of local input embeddings, apply positional embedding to the set of local input embeddings, determine a set of keys by multiplying the set of local input embeddings with a learned key matrix, determine a set of values by multiplying the set of local input embeddings with a learned value matrix, determine a set of queries by multiplying the set of local input embeddings with a learned query matrix. The node is further configured to determine a set of local memory states, receive one or more node memory states, and a set of combined node memory states. Furthermore, provide at least a portion of the set of combined node memory states to at least one next node, and determine an output based on the determined set of queries and the set of combined node memory states.

Description

[0001] NODE, METHOD, AND MULTI-AGENT INTELLIGENT NETWORK FOR PROVIDING DISTRIBUTED ATTENTION

[0002] TECHNICAL FIELD

[0003] The present disclosure relates generally to the field of wireless communication network management and, more specifically, to a node and a method for a multi-agent intelligent network providing distributed attention. Furthermore, the present disclosure relates more specifically to the multi-agent intelligent network for providing distributed attention, such as by computing distributed linear attention.

[0004] BACKGROUND

[0005] In rapidly advancing domain of communication systems, upcoming future communication systems are anticipated to heavily rely on the cooperation of distributed agents that are used to combine the processing compatibility of each of the intelligent agents and produce results. Moreover, each of the distributed agents is used to jointly process multimodal data (e.g., images, videos, or sensing data collected by each of the distributed agents) to adapt to the changing environments, solve various complex tasks, and share obtained data with other distributed agents, such as by distributing all relevant data (i.e., tokens) across the distributed agents.

[0006] Conventional communication systems, such as conventional self-attention mechanisms, cannot be computed in a distributed manner and require all the relevant data (i.e., the tokens) to be transmitted to a central server for joint computation. On the other hand, the transmission of the relevant data from each of the distributed agents to the central server is not feasible due to limitations of communication bandwidth and privacy constraints. Moreover, certain attempts have been made to overcome the communication bandwidth and privacy constraints, such as by using the self-attention of a transformer, ring attention, retentive networks, message-passing neural networks, and the like. However, such attempts fail due to many reasons, such as the unavailability of the relevant data in a single place, complex computational costs, restrictive data flow, and the like. Thus, there exists a technical problem of how to process the relevant data with minimum communication overhead while eliminating the need for any centralized coordination during the data transmission among each of the distributed agents.

[0007] Therefore, in light of the foregoing discussion, there exists a need to overcome the aforementioned drawbacks associated with the conventional nodes and conventional methods for providing distributed attention.

[0008] SUMMARY

[0009] The present disclosure provides a node of a multi-agent intelligent network and a method for the node of a multi-agent intelligent network for providing distributed attention. The present disclosure provides a solution to the existing problem of how to process the relevant data with minimum communication overhead while eliminating the need for any centralized coordination during the data transmission among each of the distributed agents. An objective of the present disclosure is to provide a solution that overcomes at least partially the problems encountered in the prior art and provides the node of the multiagent intelligent network and the method for the node of the multi-agent intelligent network for providing distributed attention.

[0010] One or more objectives of the present disclosure are achieved by the solutions provided in the enclosed independent claims. Advantageous implementations of the present disclosure are further defined in the dependent claims.

[0011] In one aspect, the present disclosure provides a node of a multi-agent intelligent network providing distributed attention, the multi-agent intelligent network comprising the node and one or more other nodes. The node is configured to receive a set of local input embeddings, apply positional embedding to the set of local input embeddings, determine a set of keys by multiplying the set of local input embeddings with a learned key matrix, determine a set of values by multiplying the set of local input embeddings with a learned value matrix, and determine a set of queries by multiplying the set of local input embeddings with a learned query matrix. Moreover, the node is characterized in that the node is further configured to determine a set of local memory states based on the set of local input embeddings, a learned key matrix, and a learned value matrix, receive one or more node memory states from at least one of the one or more other nodes, the received one or more node memory states forming a set of node memory states, determine a set of combined node memory states based on the received set of node memory states and the set of local memory states, provide at least a portion of the set of combined node memory states to at least one next node, and determine an output based on the determined set of queries and the set of combined node memory states.

[0012] Advantageously, the node of the multi-agent intelligent network is used to provide distributed attention. The node is configured to process and attend the data received from other nodes without direct data sharing, such as by processing local input embeddings and exchanging memory states, due to which the sensitive information is not directly shared between the nodes, thereby the data privacy and the data security is maintained. Furthermore, the node is configured to reduce the requirement for extensive data transmission by sharing the memory states instead of the raw input data with the one or more nodes, leading to a minimized communication overhead and an improved and efficient bandwidth utilization. Furthermore, the addition of new nodes without any changes to the existing network structure allows one or more nodes to be accommodated in a large multiagent network. In addition, the node is configured to process the data locally to provide reduced computation and latency, which is the overall time required by the node for data processing. Moreover, each node within the multi-agent network is configured to share processed insights through memory states, enabling collaborative problem-solving and an integrated approach to handling complex computations with minimal disruption, ensuring the continuity of operations.

[0013] In another aspect, the present disclosure provides a method for a node of a multi-agent intelligent network providing distributed attention, the multi-agent intelligent network comprising the node and one or more other nodes. The method comprising receiving a set of local input embeddings applying positional embedding to the set of local input embeddings, determining a set of keys by multiplying the set of local input embeddings with a learned key matrix, determining a set of values by multiplying the set of local input embeddings with a learned value matrix, determining a set of queries by multiplying the set of local input embeddings with a learned query matrix, wherein the node is characterized in that the node is further configured to determine the set of local memory states based on the set of local input embeddings, a learned key matrix and a learned value matrix, receiving one or more node memory states from at least one of the one or more other nodes, the received one or more node memory states forming a set of node memory states, determining a set of combined node memory states based on the received set of node memory states and the set of local memory states, providing at least a portion of the set of combined node memory states to at least one next node, and determining an output based on the determined set of queries and the set of combined node memory states.

[0014] The method achieves all the advantages and technical effects of the node of the present disclosure.

[0015] In yet another aspect, there is provided a multi-agent intelligent network providing distributed attention, the multi-agent intelligent network comprising a node and one or more other nodes, the multi-agent intelligent network being configured to, for the node, receive a set of local input embeddings, apply positional embedding to the set of local input embeddings, determine a set of keys, the keys being a learned key matrix, determine a set of values being a learned value matrix, determine a set of queries being a learned query matrix and the multi-agent intelligent network is characterized in that the multi-agent intelligent network is further configured to, for the node, determine a set of local memory states, receive a set of node memory states from the one or more other nodes, determine a set of combined node memory states and is determined as the sum provide at least a portion of the set of combined node memory states to at least one next node, and determine an output. Advantageously, the multi-agent intelligent network provides distributed attention with an efficient and reduced communication overhead. The multi-agent intelligent network receives the set of input embeddings from the one or more nodes without directly sharing raw data in order to ensure that the sensitive information is not directly shared between nodes for maintaining the data privacy and data security. By sharing memory states instead of raw input data, the multi-agent intelligent network is used to reduce the need for extensive data transmission, leading to minimized communication overhead and improved bandwidth utilization. The multi-agent intelligent network provides a scalable and flexible accommodation of new nodes without requiring any changes to an existing network structure in order to provide seamless expansion and integration of additional nodes. In addition, each node processes data locally and in parallel, which reduces the overall computation time and latency of the data along with the enhanced efficiency and responsiveness of the multi-agent intelligent network. The use of learned matrices (e.g., key, value, and query matrices) enables efficient parallel training on a server with large datasets, and once trained, the model parameters can be easily deployed across distributed nodes to ensure continuity of operations while handling complex computations with minimal disruption.

[0016] It is to be appreciated that all the aforementioned implementation forms can be combined.

[0017] It has to be noted that all devices, elements, circuitry, units, and means described in the present application could be implemented in the software or hardware elements or any kind of combination thereof. All steps which are performed by the various entities described in the present application, as well as the functionalities described to be performed by the various entities are intended to mean that the respective entity is adapted to or configured to perform the respective steps and functionalities. Even if, in the following description of specific embodiments, a specific functionality or step to be performed by external entities is not reflected in the description of a specific detailed element of that entity that performs that specific step or functionality, it should be clear for a skilled person that these methods and functionalities can be implemented in respective software or hardware elements or any kind of combination thereof. It will be appreciated that features of the present disclosure are susceptible to being combined in various combinations without departing from the scope of the present disclosure as defined by the appended claims.

[0018] Additional aspects, advantages, features, and objects of the present disclosure would be made apparent from the drawings and the detailed description of the illustrative implementations construed in conjunction with the appended claims that follow.

[0019] BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The summary above, as well as the following detailed description of illustrative embodiments, is better understood when read in conjunction with the appended drawings. For the purpose of illustrating the present disclosure, exemplary constructions of the disclosure are shown in the drawings. However, the present disclosure is not limited to specific methods and instrumentalities disclosed herein. Moreover, those in the art will understand that the drawings are not to scale. Wherever possible, like elements have been indicated by identical numbers.

[0021] Embodiments of the present disclosure will now be described, by way of example only, with reference to the following diagrams wherein:

[0022] FIG. 1 is a block diagram that illustrates a multi-agent intelligent network providing distributed attention, in accordance with an embodiment of the present disclosure;

[0023] FIG. 2 is a block diagram of a node in a multi-agent intelligent network providing distributed linear attention, in accordance with an embodiment of the present disclosure;

[0024] FIG. 3 is a flowchart of a method for a node of a multi-agent intelligent network, in accordance with an embodiment of the present disclosure; FIG. 4 is a diagram that illustrates an exemplary scenario for computing of a linear attention between nodes, in accordance with an embodiment of the present disclosure;

[0025] FIG. 5 is a diagram that illustrates an exemplary computing of distributed linear attention, in accordance with an embodiment of the present disclosure;

[0026] FIG 6 is a diagram that illustrates an exemplary scenario of a multipath topology for computing distributed linear attention , in accordance with an embodiment of the present disclosure;

[0027] FIG. 7 is a diagram that illustrates the training of a linear attention module in a parallel mode on powerful servers using large training datasets, in accordance with an embodiment of the present disclosure;

[0028] FIG. 8 is a diagram that illustrates an exemplary scenario of a cyclic network topology for computing distributed linear attention, in accordance with an embodiment of the present disclosure; and

[0029] FIG. 9 is a diagram that illustrates an exemplary scenario of a vehicle monitored by a network of cameras, in accordance with an embodiment of the present disclosure.

[0030] In the accompanying drawings, an underlined number is employed to represent an item over which the underlined number is positioned or an item to which the underlined number is adjacent. A non-underlined number relates to an item identified by a line linking the non-underlined number to the item. When a number is non-underlined and accompanied by an associated arrow, the non-underlined number is used to identify a general item at which the arrow is pointing.

[0031] DETAILED DESCRIPTION OF EMBODIMENTS

[0032] The following detailed description illustrates embodiments of the present disclosure and ways in which they can be implemented. Although some modes of carrying out the present disclosure have been disclosed, those skilled in the art would recognize that other embodiments for carrying out or practicing the present disclosure are also possible.

[0033] FIG. 1 is a block diagram that illustrates a multi-agent intelligent network providing distributed attention, in accordance with an embodiment of the present disclosure. With reference to FIG. 1, there is shown a multi-agent intelligent network 100 providing distributed attention. The multi-agent intelligent network 100 includes a multi-head network model 104 and a server 108. The multi-head network model 104 includes a plurality of model layers, such as a first layer 114A, a second layer 114B, kth layer 114K, nth layer 114N, and n+1 layer 114N+1. Furthermore, the multi-agent intelligent network 100 includes system layers, such as a first system layer 116A and a second system layer 116B that further includes one or more nodes, such as a first node 118A, a second node 118B, a third node 118C, and a fourth node 118D.

[0034] There is provided the multi-agent intelligent network 100 providing distributed attention. The multi-agent intelligent network 100 provides distributed attention by allowing multiple nodes to independently process and focus on different parts of the data while communicating and collaborating with each other to provide an enhanced efficiency, robustness, and data privacy while handling complex and large-scale operations in a decentralized and efficient manner with reduced communication overhead. The multi-agent intelligent network 100 includes the multi-head network model 104 that is configured to receive a model input data 102 (i.e., a sequence of input tokens), such as a query translated into tokens and provides a model output data 106, which is a sequence of output tokens translated into a response that is further utilized for computing the distributed attention. Moreover, the server 108 includes a scheduler 110, which is configured to control, schedule, or train the multi-head network model 104 based on the information (e.g. , the information received at operationl 12), such as information on system capabilities, number or nodes, and the like. The multi-head network model 104 is deployed to compute distributed attention within the multi-agent intelligent network 100 and includes multiple model layers, such as the first layer 114A, the second layer 114B, the kth layer 114k, the nth layer 114N, and the nth+1 layer 114N+1. Moreover, each layer of the model layers includes multiple head and is configured to transmit the set of local inputs along with the memory states (i.e., partially processed tokens) to the other layer of the multi-head network model 104, such as at operation 120. For example, at operation 120A, the nth layer 114N is configured to receive the local input embeddings from the previous node and, further, at operation 120B, transmits the local input embeddings (or the partially processed tokens) to the nth+1 layer 114N+1. Moreover, the detailed transmission of the local input embeddings is given and explained in detail in FIG. 2.

[0035] Furthermore, each of the model layers of the multi-head network model 104 includes at least one system layer, such as the first system layer 116A and the second system layer 116B, that are assigned to one or more nodes. In other words, each of the node executes one or more model layers and one or more heads, such as a first head 124A, a second head 124B, a third head 124C, and a fourth head 124D, are assigned to the one or more nodes. Moreover, each of the node executes all heads of the executed model layers. For example, the first head 124A, the second head 124B, the third head 124C, and the fourth head 124D are assigned to the second node 118B. As a result, the multi-agent intelligent network 100 is configured to provide distributed attention by allowing each node within the multi-agent intelligent network 100 to determine the combined node memory state and further transmit the same to the other nodes along with the set of local memory states.

[0036] In operation, the multi-agent intelligent network 100 is configured to receive a set of local input embeddings for the node. In an example, the multi-agent intelligent network 100 is configured to receive the set of local input embeddings (e.g., Xn, XI- X3) for the first node 118A. In another example, the multi-agent intelligent network 100 is configured to receive the set of local input embeddings for the second node 118B. Similarly, the multi-agent intelligent network 100 is configured to receive the set of local input embeddings for the third node 118C and the fourth node 118D. The node (e.g., the first node 118A, the second node 118B, the third node 118C, and the fourth node 118D) is configured to receive the set of local input embeddings to initialize the data processing at the local node level. Moreover, the set of local input embeddings refers to a set of tokens that represents the information about the local data input of the one or more nodes within the multi-agent intelligent network 100. As a result, the node is configured to receive the set of local input embeddings to process and interpret the local data efficiently for further computations and decision-making processes within the multi-agent intelligent network 100.

[0037] Furthermore, the multi-agent intelligent network 100 is configured to apply positional embedding to the set of local input embeddings for the node. The positional embedding, which is applied to the set of local input embeddings for the node is configured to provide the information about the position of each input element in order to identify the sequence order to provide accurate and context-aware predictions. The multi-agent intelligent network 100 calculates positional embeddings for each position in the input sequence. In an implementation, the positional embeddings can be pre-defined (e.g., sinusoidal functions) or learned during training. Thereafter, the calculated positional embeddings are added to the corresponding local input embeddings. As a result, the positional embeddings are used to capture the order of input elements to allow the multi-agent intelligent network 100 to handle a wide range of sequence lengths and structures. Additionally, the integration of positional information allows the multi-agent intelligent network 100 to handle complex patterns and relationships, providing an accurate and flexible approach to processing sequential data.

[0038] Furthermore, the multi-agent intelligent network 100 is configured to determine a set of keys, Kn, as Kn= WK Xn, WK being a learned key matrix WK. In an implementation, the set of local input embeddings (i.e., token Xn) are multiplied with the set of learned key matrix (i.e., WK) in order to determine the set of keys, for example, as shown in the equation (1) given below: -

[0039] The determination of the set of keys involves transforming the set of local input embeddings by using a learned key matrix to provide the set of keys that are further utilized to compute the memory states. Moreover, by using a learned matrix enables the multi-agent intelligent network 100 to provide an efficient and effective representation of the data that can be used for computing the linear attention.

[0040] Furthermore, the multi-agent intelligent network 100 is configured to determine a set of values, Vn, as Vn= Wv Xn, Wv being a learned value matrix, Wv. The set of values is used to encode the information that is used to compute the combined memory states that are further utilized to compute the linear computation. In an implementation, the set of local input embeddings (i.e., token Xn) are multiplied with the set of learned value matrix (i.e., Wv) in order to determine the set of keys, for example, as shown in the equation (2) given below: -

[0041] Moreover, the learned value matrix allows the encoding of the set of input embeddings into the set of learned values that are used to capture relevant information to handle varying input sizes and complexities without a significant increase in computational cost, such as by further computing the combined memory states. As a result, the determination of the set of values is used to improve the efficiency, scalability, and overall performance of the node while minimizing the communication overhead within the multi-agent intelligent network 100.

[0042] Furthermore, the multi-agent intelligent network 100 is configured to determine a set of queries, Qn, as Qn =WQ „, WQ being a learned query matrix, WQ, for the node. The set of values is used to encode the information that is used to compute the combined memory states that are further utilized to compute the linear computation. In an implementation, the set of local input embeddings (i.e., token Xn) are multiplied with the set of learned queries matrix (i.e., Wq) in order to determine the set of keys, for example, as shown in the equation (3) given below: -

[0043] Qn = WQXn(3)

[0044] Moreover, the learned value matrix allows the encoding of the set of input embeddings into the set of learned queries that are used to capture relevant information to handle varying input sizes and complexities without a significant increase in computational cost, such as by further computing the combined memory states. As a result, the determination of the set of queries is used to improve the efficiency, scalability, and overall performance of the node while minimizing the communication overhead within the multi-agent intelligent network 100.

[0045] Moreover, the multi-agent intelligent network 100 is characterized in that the multi-agent intelligent network 100 is further configured to determine a set of local memory states S’n, as S’n = XTn(WTKWv)Xn, for the node. In an implementation, the set of local memory states refers to input partial tokens that are associated with each node that is derived through a series of matrix multiplications involving the input token and a set of learned matrices, such as the set of values, set of keys, and the set of queries. Moreover, the local memory states are configured to encapsulate the information contained in the input tokens in a manner that is suitable for further processing within the node and for sharing the same with other nodes of the multi-agent intelligent network 100. Firstly, the set of input embeddings (or tokens) are received by the node locally. Thereafter, the set of keys, the set of values, and the set of queries are determined in order to further determine the set of local memory states for each node, such as the first node 118A, the second node 118B, the third node 118C, and the fourth node 118D of the multiagent intelligent network 100. The set of local memory states allows the node to perform linear distribution of attention in order to allow information sharing within the multi-agent intelligent network 100 locally, thereby maintaining data privacy within the multi-agent intelligent network 100. As a result, the set of local memory states is determined to provide an efficient local processing, distributed attention, data privacy, and reduced communication overhead in the multi-agent intelligent network 100.

[0046] In accordance with an embodiment, the multi-agent intelligent network 100 is further configured to determine the set of local memory states S’n, as S’n = KTnVnfor the node. Each node computes the key vector (Kn) and the value vector (Vn) from the set of input embeddings by using the learned matrices. Thereafter, the key vector is multiplied by the value vector in order to determine the local memory state. Moreover, by calculating local memory states as S'n= KTnVn, the node of the multi-agent intelligent network 100 is configured to efficiently encapsulate the information from the input tokens of each node. As a result, the multi-agent intelligent network 100 is configured to ensure effective distributed data processing with an improved overall network performance.

[0047] Furthermore, the multi-agent intelligent network 100 is configured to receive a set of node memory states (Sn) from the one or more other nodes. In an example, the multi-agent intelligent network 100 is configured to receive the set of node memory states (Sn) from the first node 118A. In another example, the multi-agent intelligent network 100 is configured to receive the set of node memory states (Sn) from the first node 118A and the second node 118B. In yet another example, the multi-agent intelligent network 100 is configured to receive the set of node memory states (Sn) from the first node 118A, the second node 118B, and the third node 118C. Moreover, by receiving the set of node memory states from the one or more nodes, the node is configured to integrate external information with the local memory state of the node in order to facilitate a distributed linear attention mechanism without sharing raw data.

[0048] Furthermore, the multi-agent intelligent network 100 is configured to determine a set of combined node memory states (Sn) as Sj = Sj-i + S’n, and So is determined as the sum > S' for the node. In other words, each node in the multi-agent intelligent network 100 determines the local memory states from the set of input embeddings by using learned matrices and transmits these memory states to one or more other nodes. Finally, the received memory states are integrated with the local memory states of the node to further determine the combined memory states that are used to compute the linear attention. The set of combined node memory states is used to facilitate distributed attention in order to ensure that each node has access to a comprehensive set of node combined memory states with an efficient and effective data integration and processing with reduced communication overhead, such as by transmitting aggregated states rather than raw data. Additionally, the determination of the set of node combined memory states is used to maintain data privacy, such as by sharing the combined node memory state with an improved overall performance of the multi-agent intelligent network 100.

[0049] In accordance with an embodiment, the multi-agent intelligent network 100 is further configured to determine the set of combined node memory states (Sn) as Sj = A(H)Sj-i + S’n, and H for the node and H is a Hadamard product and wherein A is an adaptation factor. The adaptation factor enables the adjustment of the received node memory states thereby allowing the multi-agent intelligent network 100 to scale the contributions of previous memory states based on current conditions or data characteristics dynamically. Moreover, the Hadamard product refers to an element- wise multiplication operation between two matrices or vectors of the same dimensions, where each element in the resulting matrix or vector is the product of the corresponding elements from the input matrices or vectors. As a result, the determination of the set of combined node memory states by using the Hadamard product and the adaptation factor is used to allow the multi-agent intelligent network 100 to improve the accuracy and relevance of the combined memory states in order to enhance the decision-making and data processing capabilities of the multi-agent intelligent network 100. In addition, the utilization of the Hadamard product and adaptation factor ensures an efficient and scalable data integration with reduced communication overhead and maintained data privacy. In accordance with an embodiment, the multi-agent intelligent network 100 is further configured to receive a counter (CTR=dj) for each received node memory state (S’), and So is determined as the sum > Ad’S' for the node, and A is an adaptation factor. The counter for each received memory state allows the multi-agent intelligent network 100 to track the relevance or freshness of the data from the one or more nodes. Moreover, each node is configured to determine the combined memory state by summing the products of the adaptation factor, the counter, and the memory states that may include parameters, such as the decay factor and the rotation, to modulate the influence of each state. In such an implementation, the adaptation factor A is e1Dand the > is a decay factor and > is a rotating factor. The decay factor reduces the contribution of older memory states, while the rotating factor represents the phase shift. As a result, the multi-agent intelligent network 100 is used to manage and prioritize the memory states, thereby enhancing the overall accuracy and relevance of the combined memory states. Additionally, such determination also reduces the risk of outdated information affecting current decisions and maintains flexibility and scalability within the multi-agent intelligent network 100.

[0050] Furthermore, the multi-agent intelligent network 100 is configured to provide at least a portion of the set of combined node memory states (Sn) to at least one next node for the node. In other words, the multi-agent intelligent network 100 is configured to determine the combined node memory state and further transmit the same to the next node within the multi-agent intelligent network 100. For example, the second node 118B is configured to receive at least the portion of the set of combined node memory states by the first node 118 A. As a result, by providing the combined memory states, the multi-agent intelligent network 100 is configured to adapt to changing environments and solve various challenges by leveraging the collective knowledge and insights of the nodes.

[0051] Furthermore, the multi-agent intelligent network 100 is configured to determine an output, On, as On = QnSnfor the node. By determining the output On, as On = QnSnfor the node, the multi-agent intelligent network 100 ensures that the output of each node includes both the local data (i.e., the local memory states) and the collective knowledge (i.e., the combined memory states) of the multi-agent intelligent network 100. As a result, the accuracy and relevance of the output are enhanced, leading to an improved overall network performance.

[0052] Advantageously, the multi-agent intelligent network 100 provides distributed attention with an efficient and reduced communication overhead. The multi-agent intelligent network 100 receives the set of input embeddings from the one or more nodes without directly sharing raw data in order to ensure that the sensitive information is not directly shared between nodes for maintaining the data privacy and data security. By sharing memory states instead of raw input data, the multi-agent intelligent network 100 is used to reduce the need for extensive data transmission, leading to minimized communication overhead and improved bandwidth utilization. The multi-agent intelligent network 100 provides a scalable and flexible accommodation of new nodes without requiring any changes to an existing network structure in order to provide seamless expansion and integration of additional nodes. In addition, each node processes data locally and in parallel, reduces the overall computation time and latency of the data and enhances the efficiency and responsiveness of the multi-agent intelligent network 100. The use of learned matrices (e.g., key, value, and query matrices) enables efficient parallel training on a server with large datasets, and once trained, the model parameters can be easily deployed across distributed nodes to ensure continuity of operations while handling complex computations with minimal disruption.

[0053] FIG. 2 is a block diagram that illustrates a node of a multi-agent intelligent network providing distributed attention, in accordance with an embodiment of the present disclosure. With reference to FIG. 2, there is shown a diagram 200 that illustrates a node 202 of the multi-agent intelligent network 100 (of FIG. 1 ) for providing distributed attention.

[0054] The node 202 refers to an individual device within the multi-agent intelligent network 100 that processes local data and communicates with the one or more nodes (the first node 118A, the second node 118B, the third node 118C, and the fourth node 118D of FIG. 1), such as user equipment (UEs) and the like of the multi-agent intelligent network 100. There is provided the node 202 of the multi-agent intelligent network 100 providing distributed attention. The node 202 of the multi-agent intelligent network 100 is configured to provide the distributed attention to handle high bandwidth requirements, privacy concerns, and scalability issues. By distributing the processing load across multiple nodes (i.e., the node 202), the multi-agent intelligent network 100 is configured to efficiently handle large-scale data, maintain data privacy, and reduce communication overhead.

[0055] In operations, the multi-agent intelligent network 100 includes the node 202 and the one or more other nodes, and the node 202 is configured to receive a set of local input embeddings. The node 202 is configured to receive the set of local input embeddings to initialize the data processing at the local node level. Moreover, the set of local input embeddings, for example, a first local input embedding 204A, a second local input embedding 204B, and a third local input embedding 204C, refers to a set of tokens that represents the information about the local data input of the one or more nodes within the multi-agent intelligent network 100. As a result, the node 202 is configured to receive the set of local input embeddings to process and interpret the local data efficiently for further computations and decision-making processes within the multi-agent intelligent network 100.

[0056] Furthermore, the node 202 is configured to apply positional embedding to the set of local input embeddings. In other words, at operation 206, the node 202 is configured to append positional embeddings to input tokens. The application of the positional embedding to the set of local input embeddings is used to encode the position of each input element in the sequence, allowing the node 202 to distinguish between different positions. In an implementation, the node 202 is configured to generate the positional embeddings, such as by using predefined or learned functions. Thereafter, the generated positional embeddings are added to the local input embeddings. As a result, by incorporating positional embedding, the node 202 is configured to distinguish between different positions of the one or more nodes, thereby allowing the node 202 to handle various types of sequential data effectively, making the multi-agent intelligent network 100 adaptable to different applications, such as text, audio, and video processing.

[0057] Furthermore, the node 202 is configured to determine a set of keys by multiplying the set of local input embeddings with a learned key matrix 208. In an implementation, the set of local input embeddings, such as the first local input embedding 204A, the second local input embedding 204B, and the third local input embedding 204C, are multiplied with learned key matrix 208 in order to obtain the set of keys, such as a first key 208A, a second key 208B, and a third key 208C. Moreover, the determination of the set of keys involves transforming the set of local input embeddings by using the learned key matrix 208 to provide the set of keys that are further utilized to compute the memory states. Moreover, by using the learned key matrix 208, node 202 is configured to provide an efficient and effective representation of the data that can be used for computing the linear attention.

[0058] Furthermore, the node 202 is configured to determine a set of values by multiplying the set of local input embeddings with a learned value matrix 210 and a set of queries by multiplying the set of local input embeddings with the learned query matrix 212. In an example, the set of local input embeddings, such as the first local input embedding 204A, the second local input embedding 204B, and the third local input embedding 204C, are multiplied with learned value matrix 210 in order to obtain the set of values, such as a first value 210A, a second value 210B, and a third value 210C. Similarly, the set of local input embeddings, such as the first local input embedding 204A, the second local input embedding 204B, and the third local input embedding 204C, are multiplied with learned query matrix 212 to determine the set of queries that includes a first query 212A, a second query 212B, and a third query 212C. The determination of the set of queries and the set of values are used to improve the efficiency, scalability, and overall performance of the node 202 while minimizing the communication overhead within the multi-agent intelligent network 100. Moreover, the node 202 is characterized in that the node 202 is further configured to determine a set of local memory states based on the set of local input embeddings, a learned key matrix 208, and a learned value matrix 210. In an implementation, the set of local memory states refers to input partial tokens that are associated with each node that is derived through a series of matrix multiplications involving the input token and a set of learned matrices, such as the set of values, set of keys, and the set of queries, at operation 216. Moreover, the local memory states, for example, a first local memory state 218A, a second local memory state 218B, and a third local memory state 218C, are configured to encapsulate the information contained in the input tokens in a manner that is suitable for further processing within the node 202 and for sharing the same with other nodes of the multi-agent intelligent network 100. Firstly, the set of input embeddings (or tokens) are received by the node locally. Thereafter, the set of keys, the set of values, and the set of queries are determined in order to further determine the set of local memory states for each node of the multi-agent intelligent network 100, such as based on the set of local embeddings, the set of learned key matrix 208, the learned value matrix 210. The set of local memory states allows the node 202 to perform attention distribution in order to allow information sharing within the multi-agent intelligent network 100 locally, thereby maintaining the data privacy within the multi-agent intelligent network 100. As a result, the set of local memory states is determined to provide an efficient local processing, distributed attention, data privacy, and reduced communication overhead in the multi-agent intelligent network 100.

[0059] In accordance with an embodiment, the node 202 is further configured to determine the set of local memory states by multiplying the set of local input embeddings with the learned key matrix 208 and with the learned value matrix 210. By using learned matrices, the node 202 is configured to transform the raw input embeddings into a format that highlights the key features and values in order to integrate and share such information with other nodes in the multi-agent intelligent network 100. By determining the local memory states by the multiplication of the set of input embeddings with the learned key and the learned value matrices, the multi-agent intelligent network 100 is configured to ensure that each node accurately encodes and processes the local data for efficient transformation of the raw data into meaningful representations, facilitating effective distributed processing and information sharing.

[0060] In accordance with an embodiment, the node 202 is further configured to determine the set of local memory states by multiplying a transpose of the set of local input embeddings with a transpose of the learned key matrix 208 and with the learned value matrix 210 and with the set of local input embeddings. By determining the set of local memory states, the node 202 is configured to capture the detailed information from the input data for identifying and encoding higher-order interactions, leading to a robust and informative memory state. Additionally, the node 202 is configured to enhance the ability of the multiagent intelligent network 100 to perform complex computations.

[0061] In accordance with an embodiment, the node 202 is further configured to determine the set of local memory states by multiplying the set of keys with the set of values. Each node computes the key vector (Kn) and the value vector from the set of input embeddings by using the learned matrices. Thereafter, the key vector is multiplied by the value vector in order to determine the local memory state. Moreover, by calculating local memory states as S'n = KTnVn, the multi-agent intelligent network 100 is configured to efficiently encapsulate the information from the input tokens of each node, such as the node 202. As a result, the node 202 is configured to ensure effective distributed data processing with an improved overall network performance.

[0062] In accordance with an embodiment, the node 202 is further configured to determine the set of local memory states by multiplying a transpose of the set of keys with the set of values. Each node computes the key vector (Kn) and the value vector ( Vn) from the set of input embeddings by using the learned matrices. Thereafter, the key vector is multiplied by the value vector in order to determine the local memory state. Moreover, by calculating local memory states as S'n = KTnVn, the node 202 of the multi-agent intelligent network 100 is configured to efficiently encapsulate the information from the input tokens of each node. As a result, the multi-agent intelligent network 100 is configured to ensure effective distributed data processing with an improved overall network performance.

[0063] Furthermore, the node 202 is configured to receive one or more node memory states from at least one of the one or more other nodes that received one or more node memory states, forming a set of node memory states. By receiving the set of node memory states from the at least one or more other nodes, the node 202 is configured to integrate external information with the local memory state of the node in order to facilitate a distributed linear attention mechanism without sharing raw data.

[0064] Furthermore, node 202 is configured to determine a set of combined node memory states based on the received set of node memory states and the set of local memory states. In other words, each node of the multi-agent intelligent network 100 determines the local memory states from the set of input embeddings by using learned matrices and transmits the memory states to one or more other nodes. Finally, the received memory states are integrated with the local memory states of the node to further determine the combined memory states that are used to compute the linear attention. The combined memory state 224 is scaled and rotated, which is added to the first memory state 218 A, for example, at operation 228 A in order to obtain a combined memory state 230A. Furthermore, the combined memory state 230A is scaled and rotated and is further add to the second memory state 218B to obtain a combined memory state 230B, for example, at operation 228B. Thereafter, the combined memory state 230B is scaled and rotated to further add with the third memory state 218C, for example, at operation 228C, to obtain a combined memory state 230C. Hence, the combined memory state 230C includes information from the combined memory state 224, 218A, 218B, and 218C. As a result, the set of combined node memory states is used to facilitate distributed attention in order to ensure that each node has access to a comprehensive set of node combined memory states with an efficient and effective data integration and processing with reduced communication overhead, such as by transmitting aggregated states rather than raw data. Additionally, the determination of the set of node combined memory states is used to maintain data privacy, such as by sharing the combined node memory state with an improved overall performance of the multi-agent intelligent network 100.

[0065] In accordance with an embodiment, the node 202 is further configured to determine the set of combined node memory states based on the received set of node memory states and the set of local memory states by determining an initial combined node memory state based on the received set of node memory states and recursively summing up the received node memory states starting with the initial combined node memory state step-wise adding the local memory states. By starting with an initial combined memory state and further incorporating the received combined memory states and the local memory states, the node 202 can effectively build a comprehensive representation of the data in the multi-agent intelligent network 100 in order to reduce the computational overhead and communication costs in order to provide an efficient and flexible multi-agent intelligent network 100 capable of handling diverse and dynamic data environments.

[0066] In accordance with an embodiment, each received node memory state is adapted by being subjected to an adaptation factor when being summed up, the adaptation factor including being weighted with a decay factor. The decay factor reduces the contribution of older memory states, while the rotating factor represents the phase shift. As a result, the node 202 of the multiagent intelligent network 100 is used to manage and prioritize the memory states, thereby enhancing the overall accuracy and relevance of the combined memory states. Additionally, such determination also reduces the risk of outdated information affecting current decisions and maintains flexibility and scalability within the multi-agent intelligent network 100.

[0067] In accordance with an embodiment, each received node memory state is adapted by being subjected to an adaptation factor when being summed up, and the adaptation factor includes being weighted with a decay factor and being rotated with a rotating factor. The decay factor attenuates the contributions of previous memory states, effectively reducing the influence over time or distance, which helps to prioritize more recent or relevant information (or the node memory states received from the one or more nodes). Moreover, the rotating factor refers to a phase shift that enables the temporal modulation in order to align or misalign with the phases of memory states, which is further used in applications involving periodic or cyclical data patterns. As a result, the adaptation factor allows the multi-agent intelligent network 100 to control the attenuation and phase shift of the combined memory states in order to prioritize the received memory states and improve the overall decision-making and processing efficiently and effectively. In accordance with an embodiment, each previous combined node memory state is weighted with a decay factor when being summed up. Weighting each previous combined node memory state with a decay factor ensures that the influence of older memory states diminishes over time. Therefore, such implementation is used to prevent the outdated information from having an undue impact on the current state of the node 202, thereby allowing the node 202 to prioritize more recent data. When summing up the combined node memory states, a decay factor is applied to each previous state that is used to optimize the performance of the multi-agent intelligent network 100.

[0068] In accordance with an embodiment, each previous combined node memory state is rotated with a rotating factor when being summed up. The rotating factor for each previous combined node memory state provides a transformation that can encode additional information, such as temporal or spatial context, into the memory state to allow an improved data integration for improving the overall performance and adaptability of the multi-agent intelligent network 100.

[0069] In accordance with an embodiment, each previous combined node memory state is weighted with a decay factor and rotated with a rotating factor when being summed up. The decay factor ensures that outdated information has less influence, while the rotating factor adds contextual richness to the memory states. Moreover, when summing up the combined node memory states, each previous state is first weighted with the decay factor to reduce the influence of the previous node memory state, and the rotating factor is used to encode additional context. Therefore, the weighted and rotated states are summed up to form the final combined memory state in order to provide an accurate, context-aware, and adaptable multi-agent network 100.

[0070] In accordance with an embodiment, the node 202 is further configured to determine the initial combined node memory state based on the received set of node memory states by summing up the received node memory states in the received set of node memory states. The node 202 is configured to collect the received set of node memory states from other nodes. Furthermore, the node 202 is configured to sum the associated memory states together to form an initial combined node memory state in order to provide a comprehensive representation of the information received from the one or more nodes.

[0071] In accordance with an embodiment, the node 202 is further configured to receive a counter for each of the received one or more node memory states, adapt the adapting factor based on the received counter, and provide a counter of the node when providing the set of combined node memory states. The counter for each received memory state allows the multi-agent intelligent network 100 to track the relevance or freshness of the data from the one or more nodes. For example, a first counter 220A and a second counter 220B are received by the node 202, which are the combined memory states of the previous nodes, such as at operation 222. At operation 224, the received memory states are combined linearly. Moreover, each node is configured to determine the combined memory state by summing the products of the adaptation factor, the counter, and the memory states that may include parameters like decay and rotation to modulate the influence of each state. In such an implementation, the node 202 is further configured to adapt the adapting factor based on the received counter by raising the adaptation factor to the power of the counter. The adaption of the adapting factor based on the received counter by raising the adaptation factor to the power of the counter enables the multi-agent intelligent network 100 to manage and prioritize the memory states, thereby enhancing the overall accuracy and relevance of the combined memory states. Additionally, such determination also reduces the risk of outdated information affecting current decisions and maintains flexibility and scalability within the multi-agent intelligent network 100.

[0072] Furthermore, the node 202 is configured to provide at least a portion of the set of combined node memory states to at least one next node. In other words, the multi-agent intelligent network 100 is configured to determine the combined node memory state and further transmits portion 232 of the set of combined node memory states to the at least one next node within the multiagent intelligent network 100. As a result, by providing the combined memory states, the multi-agent intelligent network 100 is configured to adapt to changing environments and solve various challenges by leveraging the collective knowledge and insights of the nodes. Furthermore, the node 202 is configured to determine an output based on the determined set of queries and the set of combined node memory states. The output (On) is determined as On = QnSnfor the node 202. The multi-agent intelligent network 100 ensures that the output of each node includes both the local data (i.e., the local memory states) and the collective knowledge (i.e., the combined memory states) of the multi-agent intelligent network 100. As a result, the determined output is enhanced, leading to an improved overall network performance.

[0073] In accordance with an embodiment, the node 202 is further configured to determine the output by multiplying the determined set of queries with the set of combined node memory states. By multiplying the set of combined memory states, such as at operation 234, enables the node 202 to generate the output that reflects the interaction between the one or more nodes that facilitates the extraction of relevant information for further processing. In accordance with an embodiment, the node 202 is further configured to process the output by applying a function 236 to the output. After generating the initial output by multiplying queries with combined node memory states, the node 202 is configured to apply a predefined function 236 to the output. In an implementation, the function is a non-linear activation function, normalization, or any other function that is used for further use or analysis. As a result, the application of the function to the output enhances the flexibility and applicability of the output data in order to improve the overall effectiveness of the data handling and decision-making processes within the multi-agent intelligent network 100. In accordance with an embodiment, the node 202 is further configured to apply a learned output matrix to the output, providing an output token. For example, the node 202 is configured to apply the learned output matrix to the output to obtain the output token (or a local output), such as a first output token 238A, a second output token 238B, and a third output token 238C. Moreover, the utilization of the learned output matrix allows the node 202 to generate output tokens that are optimized based on previous training in order to improve the accuracy and relevance of the output.

[0074] Advantageously, the node 202 of the multi-agent intelligent network 100 is used to provide distributed attention. The node 202 is configured to process the data received from other nodes without direct data sharing, such as by processing local input embeddings and exchanging memory states due to which the sensitive information is not directly shared between the nodes, thereby, the data privacy and the data security is maintained. Furthermore, the node 202 is configured to reduce the requirement for extensive data transmission by sharing the memory states instead of the raw input data with the one or more nodes, leading to a minimized communication overhead and an improved efficient bandwidth utilization. Furthermore, the addition of new nodes without any changes to the existing network structure allows the one or more nodes to be accommodated in a large multiagent network. In addition, the node 202 is configured to process the data locally to provide reduced computation and latency, which is the overall time required by the node for data processing. Moreover, each node within the multi-agent intelligent network 100 is configured to share processed insights through memory states, enabling collaborative problem-solving and an integrated approach to handling complex computations with minimal disruption, ensuring the continuity of operations.

[0075] FIG. 3 is a flowchart of a method for a node of a multi-agent intelligent network, in accordance with an embodiment of the present disclosure. FIG. 3 is described in conjunction with elements from FIG. 1. With reference to FIG. 3, there is shown a flowchart of method 300 that includes steps 303 to 318. The node (i.e., the node 202 of the FIG. 2) is configured to execute the method 300.

[0076] There is provided the method 300 for the node 202 of the multi-agent intelligent network 100 providing distributed attention. Moreover, the multi-agent intelligent network 100 includes the node 202 and the one or more other nodes. The node 202 of the multi-agent intelligent network 100 is configured to provide the distributed attention to handle high bandwidth requirements, privacy concerns, and scalability issues. By distributing the processing load across multiple nodes (i.e., the node 202), the multi-agent intelligent network 100 is configured to efficiently handle large-scale data, maintain data privacy, and reduce communication overhead. At step 302, the method 300 includes receiving a set of local input embeddings. At step 304, the method 300 includes applying positional embedding to the set of local input embeddings, and at step 306, the method 300 includes determining a set of keys by multiplying the set of local input embeddings with a learned key matrix. Furthermore, at step 308, the method 300 includes determining a set of values by multiplying the set of local input embeddings with a learned value matrix, and at step 310, the method 300 includes determining a set of queries by multiplying the set of local input embeddings with a learned query matrix and the method 300 is characterized in that the method 300 includes the node for determining a set of local memory states based on the set of local input embeddings, a learned key matrix and a learned value matrix. At step 312, the method 300 includes receiving one or more node memory states from at least one of the one or more other nodes, the received one or more node memory states forming a set of node memory states. Furthermore, at step 314, the method 300 includes determining a set of combined node memory states based on the received set of node memory states and the set of local memory states, and at step 316, the method 300 includes providing at least a portion of the set of combined node memory states to at least one next node. Finally, at step 318, the method 300 includes determining an output based on the determined set of queries and the set of combined node memory states.

[0077] Advantageously, the method 300 is used for providing the distributed attention in multi-agent intelligent network 100. The method 300 is used to process the data received from other nodes without direct data sharing, such as by processing local input embeddings and exchanging memory states, due to which the sensitive information is not directly shared between the nodes and the data privacy is maintained. Furthermore, the method 300 is used to reduce the requirement for extensive data transmission by sharing the memory states instead of the raw input data with the one or more nodes, leading to a minimized communication overhead and an improved efficient bandwidth utilization. Furthermore, the addition of new nodes without any changes to the existing network structure allows the one or more nodes to be accommodated in a large multi-agent network. In addition, the method 300 is used to process the data locally to reduce the computational cost and latency. Moreover, each node within the multi-agent intelligent network 100 is configured to share processed insights through memory states, enabling collaborative problem-solving and an integrated approach to handling complex computations with minimal disruption, ensuring the continuity of operations.

[0078] The steps 303 to 318 are only illustrative, and other alternatives can also be provided where one or more steps are added, one or more steps are removed, or one or more steps are provided in a different sequence without departing from the scope of the claims herein.

[0079] There is further provided a computer program product comprising program instructions for performing the method 300 when executed by one or more processors in the multi-agent intelligent network 100. The computer program product is implemented as an algorithm, embedded in a software stored in a non-transitory computer-readable storage medium. The non-transitory computer-readable storage means may include, but are not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. Examples of implementation of computer-readable storage medium, but are not limited to, Electrically Erasable Programmable Read-Only Memory (EEPROM), Random Access Memory (RAM), Read Only Memory (ROM), Hard Disk Drive (HDD), Flash memory, a Secure Digital (SD) card, Solid-State Drive (SSD), a computer-readable storage medium, and / or CPU cache memory.

[0080] FIG. 4 is a diagram that illustrates an exemplary scenario for computing a distributed attention between nodes, in accordance with an embodiment of the present disclosure. FIG. 4 is described in conjunction with elements from FIG. 1 to 2. With reference to FIG. 4, there is shown a diagram 400 that illustrates an exemplary scenario for computing a distribution attention between nodes. In an exemplary scenario, the first node 118A is configured to process the input tokens, such as a first input token 402A and a second input token 402B, while attending to the input tokens of the second node 118B, such as a first input token 404A and a second input token 404B without any direct access to the input tokens, such as through an efficient parallel training that provides high performance along with communication-efficiency and flexibility. The model parameters, such as the learned query matrix (WQ), the learned key matrix (WK), the learned value matrix (Wv), the rotating factor, and the decay factor, are obtained through prior setup, such as through the previous nodes of the multi-agent intelligent network 100 that are the same for all the nodes of the multi-agent intelligent network 100.

[0081] FIG. 5 is a diagram that illustrates an exemplary computing of a distributed linear attention, in accordance with an embodiment of the present disclosure. FIG. 5 is described in conjunction with elements from FIG. 1 to 4. With reference to FIG. 5, there is shown a diagram 500 of the that illustrates a complete procedure for computing the linear attention.

[0082] In an implementation scenario, the node (e.g., a first node 502, a second node 504, a third node 506, a fourth node 508, and a fifth node 510) is configured to append positional embedding to input tokens (e.g., a first input token 502A, a second input token 502B, a third input token 502C, a first ) and multiplies each token (i.e., Xn) with learned matrices (i.e., WQ, WK and Wv ). Moreover, the queries (i.e., Q), keys (i.e., K), and the values (i.e., V) for each of the input tokens are multiplied by each of the keys with a corresponding value to obtain the memory state for each input token. Parallelly, the node 202 is configured to combine the states received from other devices. Finally, the node 202 is configured to utilize augmented states to compute the output. Moreover, the obtained output is processed by a non-linear function. The computation of the attention is replicated by using different versions of learned matrices (i.e., WQ, WK, and Wv), and the result is multi-head linear attention, which is equivalent to multi-head self-attention in transformers. As a result, the output from each head could be combined by another trained matrix (i.e., Wo) into one output token in order to ensure that the relevant data with minimum communication overhead while eliminating the need for any centralized coordination during the data transmission among each of the distributed agents.

[0083] FIG 6 is a diagram that illustrates an exemplary scenario of a multipath topology for computing distributed linear attention, in accordance with an embodiment of the present disclosure. FIG. 6 is described in conjunction with elements from FIG. 1 to 5. With reference to FIG. 6, there is shown a diagram 600 of that depicts an exemplary scenario of a cyclic network topology for computing distributed linear attention. In an exemplary scenario, each node, such as a first node 502, a second node 504, a third node 506, a fourth node 508, and a fifth node 510 from the one or more nodes processes an input sequence (e.g., an input local token 602, an input local token 604, an input local token 606, and an input local token 608, and an input local token 610) and determines a combined memory state (e.g., a combined memory state 602A, a combined memory state 604A, a combined memory state 606A, a combined memory state 608 A, and a combined memory state 610A) that is forwarded to the next node in the multi-agent intelligent network 100. In an example, the first node 502 is configured to determine the combined memory state 602A from the input local token 602 and is further transmitted to the fifth node 510. In another example, the second node 504 is configured to determine the combined memory state 604A from the input local token 604 and is further transmitted to the third node 506, and the third node 506 utilizes the received combined memory state 604A along with the input local token 606 to determine the combined memory state 606A, which is further transmitted to the fifth node 510. Similarly, the fourth node 508 is configured to determine the combined memory state 608A from the input local token 608, which is further transmitted to the fifth node 510. Moreover, the computation and transmission of the linear attention can be repeated multiple times, and with each round, the exchanged state is progressively updated with information from all the devices of the multi-agent intelligent network 100. For example, the fifth node 510 is configured to receive the combined memory states (i.e., the combined memory state 602 A, the combined memory state 604A, the combined memory state 606A, the combined memory state 608B, and the combined memory state 610A), such as by weighing each of the received combined memory states with a decay factor and a rotating factor, such as at operation 612 and further summing the same, such as at operation 614. Moreover, the determined combined memory state 610A of the fifth node 510 is also weighed with a decay factor and a rotating factor, such as at operation 616, and is multiplied with the input local token 610, such as at operation 618. As a result, the distributed linear attention can be computed within the multi-agent intelligent network 100 with reduced communication overhead.

[0084] FIG. 7 is a diagram that illustrates the training of a linear attention module in a parallel mode on powerful servers using large training datasets, in accordance with an embodiment of the present disclosure. FIG. 7 is described in conjunction with elements from FIG. 1 to 6. With reference to FIG. 7, there is shown a diagram 700 that depicts the training of a linear attention module in a parallel mode on powerful servers using large training datasets.

[0085] In an implementation, a large training dataset 702 is provided a centralized offline training, such as at operation 704, in order to obtain an efficient parallel training of model parameters 706 (e.g., WQ, WK, Wv, and Wo). Moreover, the linear attention modules are trained efficiently by using servers, such as by using a large training dataset 702 in a centralized manner. As a result, the model parameters are trained parallelly by using large training datasets, such as the large training dataset 702, in a centralized manner. Moreover, once the model parameters are trained, then, after that, the trained model parameters are deployed on multiple nodes, such as shown in FIG. 8. for distributed inference. As a result, the training of the model is used to provide an enhanced and efficient multi-agent intelligent network 100 with reduced communication overhead.

[0086] FIG. 8 is a diagram that illustrates an exemplary scenario representing a cyclic network topology for computing distributed linear attention, in accordance with an embodiment of the present disclosure. FIG. 8 is described in conjunction with elements from FIG. 1 to 7. With reference to FIG. 8, there is shown a diagram 800 that depicts the exemplary scenario representing the deployment of the trained model parameters in multipath topology for computing distributed linear attention.

[0087] In an exemplary scenario, the trained model parameters are deployed in the first node 502, the second node 504, the third node 506, the fourth node 508, and the fifth node 510 in order to determine the local memory states and the combined memory states. Furthermore, the determined combined memory states are further transferred to the one or more nodes in order to compute the linear distributed computation. For example, at operation 802, the first node 502 is configured to transfer the combined memory state to the second node 504, and at operation 804, the second node 504 is configured to transfer the combined memory state to the third node 506. In another example, the third node 506 is configured to transfer the combined memory state to the fourth node 508 at operation 806 and to the first node 502, such as at operation 808. Furthermore, the fourth node 508 is configured to transfer the combined memory state to the fifth node 510 at operation 810, and the fifth node 510 is configured to transfer the combined memory state to the first node 502, such as at operation 812. As a result, the deployment of the learned weights to multiple nodes is used to reduce distributed inference and communication inference, thereby improving the overall performance of the multi-agent intelligent network 100.

[0088] FIG. 9 is a diagram that illustrates an exemplary scenario of a vehicle monitored by a network of cameras, in accordance with an embodiment of the present disclosure. FIG. 9 is described in conjunction with elements from FIG. 1 to 8. With reference to FIG. 9, there is shown a diagram 900 that depicts a vehicle 902, which is moving in direction 912 and is monitored by the multi-agent intelligent network 100 of the cameras, such as a first camera 904A, a second camera 904B, and a third camera 904C.

[0089] In an implementation, the cameras, such as the first camera 904A, the second camera 904B, and the third camera 904C, are configured to provide the data, such as the captured frames (e.g., a first frame 910A, a second frame 910B, and a third frame 910C) to the associated nodes. For example, the first camera 904A is configured to provide the data captured to the first node 502, and the second camera 904B is configured to provide the data captured to the second node 504. Similarly, the third camera 904C is configured to provide the data captured to the third node 506. Furthermore, each of the nodes transfers a last frame along with the combined memory state and the counter. For example, at operation 906, the first node 502 is configured to transfer the combined memory state along with the counter and a last frame to the second node 504, and at operation 908, the second node 504 is configured to transfer the last frame along with the combined memory state along with the counter. As a result, the multi-agent intelligent network 100 can be utilized for distributed linear attention, such as autonomous driving, distributed intelligent computing, smart manufacturing, e-health care, and the like.

[0090] Modifications to embodiments of the present disclosure described in the foregoing are possible without departing from the scope of the present disclosure as defined by the accompanying claims. Expressions such as "including", "comprising", "incorporating", "have", "is" used to describe and claim the present disclosure are intended to be construed in a non-exclusive manner, namely allowing for items, components or elements not explicitly described also to be present. Reference to the singular is also to be construed to relate to the plural. The word "exemplary" is used herein to mean "serving as an example, instance or illustration". Any embodiment described as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments or to exclude the incorporation of features from other embodiments. The word "optionally" is used herein to mean "is provided in some embodiments and not provided in other embodiments". It is appreciated that certain features of the present disclosure, which are, for clarity, described in the context of separate embodiments, may also be provided in combination in a single embodiment. Conversely, various features of the invention, which are, for brevity, described in the context of a single embodiment, may also be provided separately or in any suitable combination or as suitable in any other described embodiment of the disclosure.

Claims

CLAIMS1. A node (202) of a multi-agent intelligent network (100) providing distributed attention, the multi-agent intelligent network comprising the node (202) and one or more other nodes, the node (202) being configured to receive a set of local input embeddings, Xn, X1-X3, apply positional embedding to the set of local input embeddings, Xn, X1-X3, determine a set of keys, Kn, by multiplying the set of local input embeddings, Xn, X1-X3, with a learned key matrix, WK, (208), determine a set of values, Vn,by multiplying the set of local input embeddings, Xn, X1-X3, with a learned value matrix, Wv, (210), determine a set of queries, Qn, by multiplying the set of local input embeddings, Xn, X1-X3, with a learned query matrix, WQ, (212) wherein the node (202) is characterized in that the node (202) is further configured to determine a set of local memory states, S’n, based on the set of local input embeddings, Xn, X1-X3, a learned key matrix, WK, (208) and a learned value matrix, Wv, (210) receive one or more node memory states, S1, from at least one of the one or more other nodes, the received one or more node memory states forming a set of node memory states, Sn, determine a set of combined node memory states, Sn, based on the received set of node memory states, Sn, and the set of local memory states, S’n, provide at least a portion of the set of combined node memory states, Sn, to at least one next node and determine an output, On, based on the determined set of queries, Qn, and the set of combined node memory states, Sn.

2. The node (202) according to claim 1, wherein the node (202) is further configured to determine the output, On, by multiplying the determined set of queries, Qn, with the set of combined node memory states, Sn.

3. The node (202) according to any preceding claim, wherein the node (202) is further configured to process the output by applying a function to the output, On.

4. The node (202) according to any preceding claim, wherein the node (202) is further configured to apply a learned output matrix, Wo, to the output, WoQn, providing an output token.

5. The node (202) according to any preceding claim, wherein the node (202) is further configured to determine the set of combined node memory states, Sn, based on the received set of node memory states, Sn, and the set of local memory states, S’n, by determining an initial combined node memory state, So, based on the received set of node memory states, Sn, and recursively summing up the received node memory states, Sn, starting with the initial combined node memory state, So, stepwise adding the local memory states S’n; Sj=Sj-i+ S'.

6. The node (202) according to claim 5, wherein each previous combined node memory state, Sj, is weighted with a decay factor, y, when being summed up.

7. The node (202) according to claim 5, wherein each previous combined node memory state, Sj, is rotated with a rotating factor, cp, when being summed up.

8. The node (202) according to claim 5, wherein each previous combined node memory state, Sj, is weighted with a decay factor, y, and rotated with a rotating factor, cp, when being summed up.

9. The node (202) according to any of claims 5 to 8, wherein the node (202) is further configured to determine the initial combined node memory state, So, based on the received set of node memory states, Sn, by summing up the received node memory states, S1, in the received set of node memory states, Sn.

10. The node (202) according to claim 9, wherein each received node memory state, S’, is adapted by being subjected to an adaptation factor when being summed up, the adaptation factor including being weighted with a decay factor, y.1 1 . The node (202) according to claim 9, wherein each received node memory state, S', is adapted by being subjected to an adaptation factor when being summed up, the adaptation factor including being rotated with a rotating factor, cp.

12. The node (202) according to claim 9, wherein each received node memory state (S') is adapted by being subjected to an adaptation factor when being summed up, the adaptation factor including being weighted with a decay factor, y, and being rotated with a rotating factor, cp.

13. The node (202) according to any of claims 9 to 12, wherein the node is further configured to receive a counter, CTR, for each of the received one or more node memory states, S1, adapt the adapting factor based on the received counter, CTR, and provide a counter of the node (202) when providing the set of combined node memory states, Sn.

14. The node (202) according to claim 13, wherein the node (202) is further configured to adapt the adapting factor based on the received counter by raising the adaptation factor to the power of the counter, CTR.

15. The node (202) according to any preceding claim, wherein the node (202) is further configured to determine the set of local memory states, S’n, by multiplying the set of local input embeddings, Xn, X1-X3, with the learned key matrix, WK, (208) and with the learned value matrix, Wv, (210).

16. The node (202) according to claim 15, wherein the node (202) is further configured to determine the set of local memory states, S’n, by multiplying a transpose of the set of local input embeddings, Xn, X1-X3, with a transpose of the learned key matrix, WK, (208) and with the learned value matrix, Wv, (210) and with the set of local input embeddings, Xn, X1-X3.

17. The node (202) according to any preceding claim, wherein the node (202) is further configured to determine the set of local memory states, S’n, by multiplying the set of keys, Kn, with the set of values, Vn.

18. The node (202) according to any preceding claim, wherein the node (202) is further configured to determine the set of local memory states, S’n, by multiplying a transpose of the set of keys, Kn, with the set of values, Vn.

19. A method (300) for a node (202) of a multi-agent intelligent network (100) providing distributed attention, the multiagent intelligent network (100) comprising the node (202) and one or more other nodes, the method (300) comprising receiving a set of local input embeddings, Xn, X1-X3, applying positional embedding to the set of local input embeddings, Xn, X1-X3, determining a set of keys, Kn, by multiplying the set of local input embeddings, Xn,Xl-X3, with a learned key matrix, WK, (208), determining a set of values, Vn, by multiplying the set of local input embeddings, Xn,Xl-X3, with a learned value matrix, Wv, (210)determining a set of queries, Qn, by multiplying the set of local input embeddings, Xn,Xl-X3, with a learned query matrix, WQ, (212) wherein the method (300) is characterized in that the method further comprises the node (202) determining a set of local memory states, S’n, based on the set of local input embeddings, Xn, X1-X3, a learned key matrix, WK, (208) and a learned value matrix, Wv, (210), receiving one or more node memory states, S1, from at least one of the one or more other nodes, the received one or more node memory states forming a set of node memory states, Sn, determining a set of combined node memory states, Sn, based on the received set of node memory states, Sn, and the set of local memory states, S’n, providing at least a portion of the set of combined node memory states, Sn, to at least one next node, and determining an output, On, based on the determined set of queries, Qn, and the set of combined node memory states, Sn.

20. A computer program product comprising program instructions for performing the method (300) according to claim 19, when executed by one or more processors in a multi-agent intelligent network (100).provide at least a portion of the set of combined node memory states, Sn, to at least one next node, and determine an output, On, as On = QnSn.

22. The multi-agent intelligent network (100) according to claim 21, wherein the multi-agent intelligent network (100) is further configured to determine, for the node (202), the set of combined node memory states, Sn, as Sj = A(H)Sj-i + S’n, wherein (H) is a Hadamard product and wherein A is an adaptation factor.

23. The multi-agent intelligent network (100) according to claim 22, wherein the adaptation factor A is ye1<p, wherein y is a decay factor and cp, is a rotating factor.

24. The multi-agent intelligent network (100) according to any of claims 21 to 23, wherein the multi-agent intelligent network (100) is further configured to, for the node (202), receive a counter, CTR=dj, for each received node memory states, S', wherein So is determined as the sum E Al1-' S’ , wherein A is an adaptation factor.

25. The multi-agent intelligent network (100) according to claim 24, wherein the adaptation factor A is ye1<p. wherein y is a decay factor and cp, is a rotating factor.

26. The multi-agent intelligent network (100) according to any of claims 21 to 25, wherein the multi-agent intelligent network (100) is further configured to determine, for the node (202), the set of local memory states S’n, as S’n = KTnVn.

Images