A neural network architecture based on swarm routing and emergent memory
By introducing local neighborhood awareness and pheromone field mechanisms into the Transformer architecture, the problems of high computational complexity and memory decay in long sequence tasks are solved, and adaptive dynamic routing and steady-state memory are realized, which can adapt to multimodal and distributed agent cooperation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HANGZHOU DIANZI UNIVERSTIY INFORMATION ENG SCHOOL
- Filing Date
- 2026-03-19
- Publication Date
- 2026-06-19
AI Technical Summary
The existing Transformer architecture suffers from high computational complexity, severe memory decay, rigid routing mechanisms, and a lack of swarm intelligence in long sequence tasks, resulting in high computational resource consumption, difficulty in maintaining long-range information, and difficulty in maintaining the global semantic chain.
The system introduces a local neighborhood perception module, a pheromone field module, a proxy update unit module, a group routing module, an emergent memory unit module, a feedback regulation module, and a cross-group communication bus module to form a closed-loop logic chain. By reducing computational complexity through local neighborhood perception and pheromone field mechanisms, it achieves cross-layer chain dependency and self-organizing collaboration.
It significantly reduces the computational overhead of long sequence tasks, solves the memory decay problem, and achieves adaptive dynamic routing and steady-state memory, making it suitable for natural language, multimodal, temporal prediction, and distributed agent collaboration scenarios.
Smart Images
Figure CN122242630A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence and deep learning model structure design technology, and in particular to a neural network architecture based on population routing and emergent memory. Background Technology
[0002] The Transformer architecture is a deep learning model based on the self-attention mechanism. Since its introduction in 2017, models based on self-attention have gradually become the mainstream choice for various sequence modeling tasks due to their powerful global dependency modeling capabilities. It has revolutionized the field of Natural Language Processing (NLP) and become the cornerstone of modern models such as BERT and GPT.
[0003] With the widespread application of the Transformer architecture in natural language processing and multimodal tasks, existing models have demonstrated powerful representation capabilities, but have also gradually revealed the following shortcomings:
[0004] 1. High computational complexity: Traditional attention mechanisms require global computation of the correlation between tokens, with a complexity of O(N). 2 •d), where N represents the sequence length and d represents the feature dimension. This complexity is acceptable when processing short sequences, but in the case of long sequences, the quadratic complexity will cause the consumption of computing resources to increase exponentially, which will lead to significant computing and storage overhead.
[0005] 2. Severe memory decay: Existing hidden states or caching mechanisms mostly rely on sequential propagation. Long-range information gradually decays during inter-layer iterations. As the number of layers increases, early information will be severely decayed, making it difficult to maintain the global semantic chain.
[0006] 3. Rigid routing mechanism: Routing methods such as Mixture of Experts only work in single-step selection, lack feedback adjustment across time and across agents, and do not have true self-organizing capabilities;
[0007] 4. Lack of swarm intelligence characteristics: Ant colonies and bee colonies in nature can form globally optimal paths through local interactions and pheromone transmission. This kind of emergent intelligence mechanism has not yet been systematically introduced into deep learning neural network architecture.
[0008] Therefore, we propose a neural network architecture based on group routing and emergent memory to solve the above problems. Summary of the Invention
[0009] This invention addresses the shortcomings of existing Transformer architectures, such as high computational complexity, severe memory decay, rigid routing mechanisms, and lack of swarm intelligence, by providing a neural network architecture based on swarm routing and emergent memory.
[0010] This invention is achieved through the following technical solution:
[0011] A neural network architecture based on group routing and emergent memory includes an input embedding layer forming a closed-loop logic chain, a local neighborhood perception module, a pheromone field module, an agent update unit module, a group routing module, an emergent memory unit module, a feedback adjustment module, a cross-group communication bus module, and an output aggregation module.
[0012] The local neighborhood awareness module is used to construct a neighborhood graph for each agent node and generate neighborhood messages;
[0013] The pheromone field module is used to update the global pheromone field matrix based on the release signal of the agent node and provides a feedback signal;
[0014] The agent update unit module is used to integrate the agent state of the previous layer, neighborhood messages, pheromone feedback signals and cross-layer memory signals, and update the agent state through a gating mechanism.
[0015] The group routing module is used to calculate the routing score based on the updated agent state and pheromone feedback signal, and generate the routing output;
[0016] Emergent memory unit modules are used to write agent states, routing outputs, and pheromone feedback signals into the memory bank, and support on-demand reading to form cross-layer chain dependencies;
[0017] The feedback adjustment module is used to calculate the adjustment factor based on pheromone field statistics, memory redundancy, and routing entropy, and to adjust the pheromone field and memory writing.
[0018] The output aggregation module is used to fuse the final agent state, the emergent memory matrix, and the output of the cross-group coordination signal generation task.
[0019] In a preferred embodiment of the present invention, the local neighborhood perception module selects a K-order neighborhood set by calculating the cosine similarity between proxy nodes, and generates a neighborhood weight matrix and a neighborhood message aggregation result, wherein the aggregation complexity is approximately O(NKd), and K is much smaller than the sequence length N.
[0020] In a preferred embodiment of the present invention, the formula for selecting the K-order neighborhood is as follows: Where cos is the cosine similarity, TopK returns the set of the K indices of the candidate set in descending order of a given score. It is the output of the upper-level proxy update unit;
[0021] The formula for constructing the neighborhood weight matrix is as follows: ;
[0022] The formula for constructing neighborhood message aggregation is as follows: Furthermore, the trainable mapping of the agent update unit channel is aligned to obtain the local neighborhood elimination of agent i. , , These are the parameters that the model learns during training. The expected message dimension for the agent update unit comes from the model structure constants.
[0023] In a preferred embodiment of the present invention, the pheromone update rule of the pheromone field module is as follows: ,in Let λ represent the pheromone field matrix of the Lth layer, where λ is the attenuation coefficient. The amount of new pheromone released by all agent nodes at layer L;
[0024] When information is released, The calculation incorporates a dual coupling between the local neighborhood aggregation result and the agent's updated state. The calculation formula is as follows: , where η∈R + , which is a positive real hyperparameter for controlling the release intensity; "+" represents the set of positive real numbers. This represents the state of the layer above the i-th agent; The neighborhood messages output by the LNP at this layer; The release function maps the agent state and neighborhood messages together to the release signal.
[0025] In a preferred embodiment of the present invention, the input vector of the proxy in the proxy update unit module is first concatenated as follows: And introduce a gating mechanism The agent state is updated via gating, and the update formula is as follows: ;
[0026] in, It is a trainable parameter matrix; σ is the bias term; σ is the Sigmoid function, which ensures that the gate value is between (0,1); It is a step-by-step multiplication method; This represents the concatenation of vectors.
[0027] In a preferred embodiment of the present invention, in the group routing module at layer L, the input of each agent i is the updated state. pheromone feedback signals With neighborhood candidate set This is used to limit the candidate range for route selection;
[0028] By defining a route scoring function R(i,j), the route strength between agent i and candidate neighbor j is characterized, and the formula is as follows: ,in, ∈R d×d It is a trainable parameter matrix; Indicates routing preference based on state similarity; γ represents the global pheromone-based signal; γ represents the global pheromone term weight.
[0029] During route selection, agent i considers its candidate set N. K Neighbors are assigned routing weights in (i) The formula is The final routing output is .
[0030] In a preferred embodiment of the present invention, the input to the emergent memory unit module includes the agent state. ∈R d Routing results With the global pheromone field Φ (L) ∈R n×d ;
[0031] The emergent memory unit module uses a weighted write rule to write the communication results of the agent into the hierarchical memory. The formula is ,in ;
[0032] Simultaneously, the emergent memory unit module supports on-demand reading, allowing agents to access M in subsequent layers. (L) Extracting historical chain memory This is the memory vector to be read, used by the next-level agent update unit module. Its reading mechanism is as follows: , where Attn() represents the attention-based retrieval function.
[0033] In a preferred embodiment of the present invention, the feedback adjustment module receives global pheromone field statistics. Redundancy of memory matrix Proxy routing entropy Input signal and generate global adjustment factor β (L) Global regulation factor β (L) Through formula The calculation yielded W. f b f For training parameters, β (L) ∈(0,1).
[0034] In a preferred embodiment of the present invention, the cross-group communication bus module is used to temporarily store and merge group sets G={G1,G2,…,Gm} and group communication results in a multi-group scenario. Field signal statistics Each of these groups Includes several agents, .
[0035] Specifically, the temporary storage mechanism involves the cross-group communication bus module allocating an independent buffer slot for each group, where communication and field signals are temporarily stored. Simultaneously, the buffer representation vector of group Gg in the Lth layer is determined. With bus matrix B (L) ,in , ;
[0036] in For buffer functions, by defining W b b b To train the parameters, the group communication and field signal are uniformly encoded and stored in a buffer for subsequent cross-group alignment.
[0037] The fusion mechanism specifically involves the cross-group communication bus module aligning and merging the buffer slots of each group, calculated using the following formula: Where v is a trainable score vector, representing the group buffer. Linear scoring is performed to obtain the importance score of the group, similar to the "query vector" in the attention mechanism, but here it is a globally learnable parameter, fixed for group-level fusion. C (L) It serves as a cross-group fusion signal, acting as a cross-group input for the emergent memory unit module and the output aggregation module.
[0038] In a preferred embodiment of the present invention, the output aggregation module receives the agent state set at layer L=T. Emergent memory matrix With cross-group coordination signals The input is calculated using a formula to determine the output vector. The calculation formula is: ;
[0039] in, It is the average of the final states of all agents. It is the average of the memory matrix. It is a cross-group signal, which is uniformly mapped to the output space by splicing the group average, memory average, and cross-group signal.
[0040] The beneficial effects of this invention are:
[0041] 1. This invention constructs a sparse, locally consistent neighborhood graph through a local neighborhood perception module and generates a neighborhood aggregation representation that can be further processed by the proxy update unit, reducing the global O(N) time complexity. 2 •d) The pairwise interactions are reduced to approximately O(NKd) local interactions (where K < N), replacing global attention computation with local interactions, thereby significantly reducing computational overhead in long sequence tasks;
[0042] 2. This invention utilizes the pheromone field Φ (L) The attenuation coefficient λ and the enhancement coefficient η are combined to form an emergent memory bank M. (L) This enables the accumulation and replay of cross-layer chain dependencies, thus solving the problem of long-range information decay in traditional architectures.
[0043] 3. In this invention, the group routing module and the feedback adjustment module are used to make the routing matrix R of the Lth layer... (L) The system is adaptively updated under the control of feedback adjustment parameters γ and κ (where γ is the global weight of the pheromone term and κ represents the self-balancing parameter, and γ and κ take positive real numbers), thereby breaking through the limitation of static Mixture of Experts.
[0044] 4. This invention utilizes the neighborhood N K (i) with cross-group communication bus C (T) This enables self-organizing collaboration among agent nodes, forming an emergent globally optimal path that requires no central control.
[0045] In summary, this invention introduces two core mechanisms, "group routing" and "emergent memory," into the network layer. By utilizing the closed-loop chain logic of the local neighborhood perception module, pheromone field module, agent update unit module, group routing module, emergent memory unit module, feedback adjustment module, and output aggregation module, it achieves unified modeling of long-range dependencies, dynamic routing, and steady-state memory, making it adaptable to scenarios such as natural language, multimodal, temporal prediction, reinforcement learning, and distributed agent collaboration. Attached Figure Description
[0046] Figure 1 This is a flowchart of a neural network architecture based on group routing and emergent memory according to the present invention. Detailed Implementation
[0047] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more readily understood by those skilled in the art.
[0048] like Figure 1The neural network architecture shown consists of an input embedding layer, a local neighborhood perception module, a pheromone field module, an agent update unit module, a group routing module, an emergent memory unit module, a feedback adjustment module, a cross-group communication bus module, and an output aggregation module. Each module forms a closed-loop processing link.
[0049] The responsibility of the local neighborhood awareness module is to construct a sparse, locally consistent neighborhood graph for each token at layer L, and generate a neighborhood aggregation representation that can be further processed by the proxy update unit module. This approach reduces the global O(N) time complexity. 2 The pairwise interactions of ) are reduced to approximately O(NK) local interactions, where K is much smaller than the sequence length N. On the other hand, it provides structurally constrained and semantically stable inputs for the downstream agent update unit module → group routing module → emergent memory unit module → pheromone field module, avoiding subsequent routing and memory oscillations under noise.
[0050] The local neighborhood perception module selects its K most similar neighbors at the Lth layer to form a neighborhood set. And generate the neighborhood weight matrix. and neighborhood message aggregation result;
[0051] The formula for selecting the K-order neighborhood is as follows: Where cos is the cosine similarity, and TopK returns the set of the top K indices in the candidate set in descending order of a given score. This step belongs to existing commonly used neighbor selection methods and is only used as input preparation for this invention. It is the output of the upper-level proxy update unit;
[0052] The formula for constructing the neighborhood weight matrix is as follows: ;
[0053] Unlike existing practices, the weight matrix α (L) With adjacent structure N K (i) will be displayed as a priori for the initialization of the group routing module.
[0054] The formula for constructing neighborhood message aggregation is as follows: Furthermore, the trainable mapping of the agent update unit module channel is used to obtain the local neighborhood elimination of agent i. , , These are the parameters that the model learns during training. The expected message dimension for the proxy update unit module comes from the model structure constants;
[0055] Example: Suppose the input is a technical document, the sequence length is N=2048, and the feature dimension is d=512. In the L=1 layer, the agent i=128 has the state s. 128 (0) ∈R 512 The neighborhood set is N, calculated using TopK similarity. K (128), where K=16. Neighborhood weight matrix α (1) The result is obtained through softmax, followed by aggregation of neighborhood messages to form... ∈R 256 (Let d) m =256). This result will serve as one of the inputs to the agent update unit module.
[0056] The pheromone field module updates the global pheromone field matrix based on the release signals of agent nodes and provides feedback signals. This mechanism, introduced into the neural network, allows the interaction results of agent nodes to accumulate and propagate in the global "field," thereby achieving cross-layer chained memory and self-organizing dynamic adjustment. Unlike the attention mechanism of existing Transformers, the pheromone field module does not rely on a centralized global matrix but instead relies on the accumulation and attenuation of local agent signals to form a global distribution. This mechanism ensures that the network can maintain long-term dependency and robustness without central control.
[0057] The pheromone field module is defined as Where N is the number of proxy nodes (i.e., the number of input tokens), and the length of the source input sequence. d is the feature dimension, derived from model structure constants. R is the set of real numbers. Φ (L) Store the set of signal vectors released by all agent nodes at layer L;
[0058] The pheromone update rule of the pheromone field module is as follows: ,in Let L be the pheromone field matrix of the Lth layer. The j-th row is denoted as λ is the decay coefficient, λ∈(0,1), which is the existing technology and represents the decay of pheromones over time. This represents the amount of new pheromone released by all agent nodes at layer L;
[0059] Unlike existing technologies, this invention jointly maps neighborhood messages and agent states to a release signal during information release, defining a pheromone release function. Each agent releases a signal based on its own state and neighboring messages, as shown in the formula: ,in This represents the state of the i-th agent from the previous layer, which comes from the agent update unit module. This refers to the neighborhood messages output by the LNP at this layer. W E (L) be (L) σ represents the trainable parameters. σ represents the nonlinear activation function.
[0060] Unlike traditional pheromone updates, this invention... The calculation introduces a dual coupling between the local neighborhood aggregation result and the agent update state. This is the newly added release amount in this layer, calculated using the following formula: , where η∈R + , where "+" represents the set of positive real numbers, is a positive real number hyperparameter used to control the release intensity. In this invention... It not only depends on the node state, but also integrates the contextual information of the neighborhood structure, thus making the pheromone field module a chain-like memory carrier across nodes and layers, rather than just a numerical storage of decay accumulation.
[0061] In this invention, pheromone reading is the field signal vector fed back to agent i. The formula is ;in . This represents the vector at the j-th position in the pheromone field. Each agent reads a signal from the global pheromone field at layer L. This is for global use by the agent update unit module. To read the projection parameters. The reading weight is the value from agent i to field position j.
[0062] The logic in this module is as follows: pheromone field module → agent update unit module. The update formula for each agent will use three inputs in combination: Ensure that the agent considers both the local neighborhood and the global pheromone signal when updating.
[0063] Pheromones field module → Emergent memory unit module, global field Φ (L) It will be selectively written to the emergent memory unit: Among them, g write This is a write function defined in this invention, used to form a cross-layer chain memory.
[0064] Example: Continuing from the previous example, at level L=1, the state of agent i=128 With neighboring messages After splicing and inputting the pheromone release function f emit Receive release signal ∈R d The global field is updated as follows: .in =0.9 indicates a 10% attenuation. Subsequently, agent i=128 selectively reads the feedback signal from the global field. This serves as the input for the next step of the agent update unit module.
[0065] The agent update unit module is used to integrate the agent state of the previous layer, neighborhood messages, pheromone feedback signals and cross-layer memory signals, and update the agent state through a gating mechanism.
[0066] In the Lth layer, the input of the i-th agent in the agent update unit module includes the state of the previous layer. Output from the agent update unit module; neighborhood messages This indicates the aggregation of messages obtained by the agent after interacting with its local neighbors; pheromone field feedback. , representing global context signals; cross-layer memory signals This indicates that the emergent memory unit module selectively returns the historical chain memory.
[0067] In the agent update unit module, the agent's input vector is first concatenated as follows: And introduce a gating mechanism The agent state is updated via gating, and the update formula is as follows: ;
[0068] in, It is a trainable parameter matrix; σ is the bias term; σ is the Sigmoid function, which ensures that the gate value is between (0,1); It is a step-by-step multiplication method; This represents the concatenation of vectors.
[0069] The logic in this module is as follows: Agent update unit module → Group routing module, updated agent state. This will be passed as input to the group routing module to determine the dynamic path connection relationships between agents;
[0070] Agent update unit module → pheromone field module, new state It will again serve as the pheromone release function f emit The input affects the global pheromone field update of the next layer;
[0071] Agent update unit module → Emergent memory unit module, some states or aggregation results can be written to the emergent memory unit to form a cross-layer chain memory;
[0072] Example: Continuing with the above scenario, at level L=1, the input for agent i=128 includes: the state of the previous layer. Neighborhood messages pheromone feedback The three elements are concatenated to form a vector. After passing through the gating mechanism Then, a weighted fusion is performed with the old state to obtain the new state. This state will be passed to the swarm routing module and partially written into the pheromone field update.
[0073] The group routing module is used to calculate the routing score based on the updated agent state and pheromone feedback signal, and generate the routing output;
[0074] In layer L, the input to each agent i in the group routing module includes the updated state. (From the agent update unit module), pheromone feedback signal (From the pheromone field module) and neighborhood candidate set (From the local neighborhood awareness module), used to limit the candidate range for route selection;
[0075] By defining a route scoring function R(i,j), the route strength between agent i and candidate neighbor j is characterized, and the formula is as follows: ,in, ∈R d×d It is a trainable parameter matrix; Indicates routing preference based on state similarity; γ represents the global pheromone-based signal; γ represents the global pheromone term weight.
[0076] During route selection, agent i considers its candidate set N. K Neighbors are assigned routing weights in (i) The formula is The final routing output is The routing result Will be in relation to one's own state Fusion, as input for writing to the emergent memory unit module;
[0077] The logic in this module is as follows: Group routing module → Emergent memory unit module, routing result. As an external "communication product" of the agent, it is used by the emerging memory unit module to write cross-layer chain memory;
[0078] Group routing module → pheromone field module, routing strength distribution It can be used as an additional adjustment factor for the pheromone release function to achieve bidirectional feedback between routing and field;
[0079] Group routing module → cross-group communication bus module: In multi-group scenarios, the group routing module can establish proxy connections across groups to form inter-group communication paths.
[0080] Example: Continuing the previous scenario, at level L=1, the state of agent i=128 is... ∈R 512 The pheromone signal is ∈R512 Its neighborhood candidate set size is K=16. For neighbors j∈N 16 (128), Calculate the route score: After softmax normalization, the distribution is obtained. Then, a weighted sum is taken to form the routing result. ∈R 512 The result will be passed to the emergent memory unit module to participate in cross-layer memory writing.
[0081] The Emergent Memory Unit Module is used to write agent states, routing outputs, and pheromone feedback signals into the memory bank and supports on-demand reading to form cross-layer chain dependencies. The Emergent Memory Unit Module is one of the core innovative modules. Unlike Transformer, which only relies on the hidden state that propagates layer by layer, the Emergent Memory Unit Module provides a dynamic memory bank that crosses levels and agents, and supports the accumulation and backtracking of historical signals.
[0082] At layer L, the inputs to the emergent memory unit module include: agent state. ∈R d (From the proxy update unit module), routing results (From the group routing module), global pheromone field Φ (L) ∈R n×d (From the pheromone field module);
[0083] During input, the emergent memory unit module uses a weighted write rule to write the communication results of the agent into the hierarchical memory. The formula is ,in ;
[0084] During reading, the emergent memory unit module supports on-demand reading, and the agent can retrieve data from M in subsequent layers. (L) Extracting historical chain memory This is the memory vector to be read, used by the next-level agent update unit module. Its reading mechanism is as follows: , where Attn() represents an attention-based retrieval function;
[0085] The logic in this module is as follows: Emergent Memory Unit Module → Agent Update Unit Module, providing memory retrieval results. As one of the inputs to the next-level proxy update unit module, it enables cross-level dependency retention.
[0086] Emergent memory unit module → Feedback adjustment module: The sparsity and redundancy detection results of memory updates will be fed back to the feedback adjustment module to assist in controlling global stability.
[0087] Emergent memory unit module → Output aggregation module, the final task output can selectively incorporate the memory matrix. Improve the ability to reason in long contexts;
[0088] In this invention, the emerging memory unit module displays cross-layer memory, forming chain dependencies; the write function combines agent state, routing results and pheromone signals to ensure global and local consistency; memory can be shared by multiple layers, improving the ability to model long dependencies; the decay parameter λ controls the balance between memory retention and forgetting;
[0089] Example: The state of agent i=128 is ∈R 512 The routing output is ∈R 512 The pheromone signal is ∈R 512 After concatenation, input the following into the function: In the next layer, when L=2, agent i=128 passes through the attention function from M. (1) Reading memory vectors This is then used as an additional input to AUU, forming a cross-layer dependency.
[0090] The feedback adjustment module is used to calculate the adjustment factor based on pheromone field statistics, memory redundancy and routing entropy, and to adjust the pheromone field and memory writing. The feedback adjustment module is another key innovative module, which is mainly responsible for controlling the dynamic balance between the pheromone field module and the emergent memory unit module.
[0091] At layer L, the feedback adjustment module receives global pheromone field statistics. Redundancy of memory matrix Proxy routing entropy The input signal is simultaneously fed back to the control module, which defines a global control factor β. (L) Combining the above three indicators, the formula is: W f b f For training parameters, β (L) ∈(0,1) is the global adjustment coefficient;
[0092] The adjustment mechanism in the feedback adjustment module is: adjustment of the pheromone field. This reduces the excessive accumulation of field signals and avoids global noise.
[0093] Adjustment of memory writing, A balance is introduced between old and new memories to control redundant stacking.
[0094] Feedback on routing entropy, if If the value is too low, the exploration level will be increased in the next round of route selection (e.g., softmax temperature coefficient) to avoid over-concentration.
[0095] The logic in this module is as follows: Feedback adjustment module → Pheromones field module, the adjusted field signal is re-injected into the next layer to maintain global consistency;
[0096] Feedback adjustment module → Emergent memory unit module, updates memory writing rules, and maintains memory sparsity and effectiveness;
[0097] Feedback adjustment module → Cross-group communication bus module: In cross-group scenarios, the feedback adjustment module controls the relative weights of signals from different group fields to achieve a balance of cross-group information.
[0098] The feedback adjustment module in this invention comprehensively judges the system state by considering three indicators: field average, memory redundancy, and routing entropy. A global adjustment factor is introduced. It simultaneously acts on the pheromone field module, the emergent memory unit module, and the group routing module. This enables the invention to maintain stable evolution in deep reasoning and long sequence tasks, rather than relying on simple normalization.
[0099] Example: Continuing the previous scenario, at layer L=1, the global average value is =0.12. Memory redundancy R (1) =0.08. Routing entropy H (1) =2.45. β is obtained after inputting the feedback adjustment module function. (1) =0.73. Therefore, we get: The adjusted field signal and memory matrix are passed to the next layer to avoid noise and redundancy accumulation and maintain the diversity of routing distribution.
[0100] It also includes a cross-group communication bus module, which is used to temporarily store and merge group sets G={G1,G2,…,Gm} and group communication results in multi-group scenarios. (From the group routing module) and field signal statistics (From the pheromone field module), where each group Includes several agents, .
[0101] Specifically, the temporary storage mechanism involves the cross-group communication bus module allocating an independent buffer slot for each group, where communication and field signals are temporarily stored. Simultaneously, the buffer representation vector of group Gg in the Lth layer is determined. With bus matrix B (L) ,in , ;
[0102] in For buffer functions, by defining W b b b To train the parameters, the group communication and field signal are uniformly encoded and stored in a buffer for subsequent cross-group alignment.
[0103] The fusion mechanism specifically involves the cross-group communication bus module aligning and merging the buffer slots of each group, calculated using the following formula: ,in ∈R d ;v∈R d For trainable score vectors, the group buffer representation is used. Linear scoring is performed to obtain the importance score of the group, similar to the "query vector" in the attention mechanism, but here it is a globally learnable parameter, fixed for group-level fusion. C (L) It serves as a cross-group fusion signal, acting as a cross-group input for the emergent memory unit module and the output aggregation module.
[0104] The logic in this module is as follows: cross-group communication bus module → emergent memory unit module, cross-group signal C (L) It can be written to the global memory matrix to achieve shared memory across modalities / groups;
[0105] Cross-group communication bus module → feedback adjustment module. The buffer can serve as the monitoring interface for the feedback adjustment module to detect imbalances between groups and adjust the global adjustment factor.
[0106] Cross-group communication bus module → output aggregation module. When the final task is output, the coordination signal provided by the cross-group communication bus module serves as an important source of multi-group information fusion.
[0107] This invention introduces a buffer mechanism into the cross-group communication bus module, with the innovation lying in the combination of buffer and group-level fusion. This avoids direct interference between groups. The complexity is reduced through group-level representation, from O(N) 2 The complexity is reduced to O(mN), where m≪Nm. The fusion mechanism allows for task-driven cross-swarm coordination, which better aligns with the needs of distributed intelligence.
[0108] Example: Continuing from the previous scenario, suppose the system contains two groups:
[0109] G1: Text proxy group, containing N1=2048 tokens;
[0110] G2: Image proxy group, containing N2=1024 patches.
[0111] At layer L=1, the text group communication results are as follows: ∈R 512 Average ∈R 128Image group communication results: ∈R 512 Average ∈R 128 .
[0112] Buffer function calculation: ;
[0113] Cross-group integration: ;
[0114] final It is written into global memory for cross-modal reasoning in subsequent layers.
[0115] The output aggregation module is used to fuse the final agent state, the emergent memory matrix, and the output of the cross-group coordination signal generation task.
[0116] The output aggregation module receives the following inputs at layer L=T:
[0117] Proxy state set: , representing the hidden state of all agents at the final layer;
[0118] Emergent memory matrix Chain-like memory accumulated across layers captures long-term dependency information;
[0119] Cross-group coordination signals The multi-group fusion results from CSB ensure the consistency of information from different groups;
[0120] And calculate the output vector using the formula. The calculation formula is: ;
[0121] in, It is the average of the final states of all agents. It is the average of the memory matrix. It is a cross-group signal, which is uniformly mapped to the output space by splicing together the group average, memory average and cross-group signal;
[0122] The logic in this module is as follows: Output aggregation module → User interface. The output aggregation module is the only module exposed to the task interface. All internal details (pheromone field module, agent update unit module, group routing module, emergent memory unit module, feedback adjustment module, cross-group communication bus module) are completed internally.
[0123] Output aggregation module → Visual debugging, selective export ( , , This aids in analyzing the dynamic evolution within the model.
[0124] Example: Continuing the previous scenario, suppose the task is to categorize a document with N tokens (e.g., N=2048):
[0125] The average state of the final layer proxy is
[0126] The average memory matrix is
[0127] Cross-group coordination signal is ∈R 512 .
[0128] After the three are concatenated, the following is processed by the fusion function: The output vector z is obtained. (T) ∈R 512 Under the category interface If the output dimension is 10, it means that the document is classified into one of the 10 categories.
[0129] This invention integrates agent group state, emergent memory, and cross-group coordination signals into a triple information source, resulting in more stable output that can cover both short-term and long-term dependencies. This ensures information integrity and coordination in multimodal and multi-group tasks.
[0130] It should be noted that the above embodiments only illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be pointed out that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.
Claims
1. A neural network architecture based on population routing and emergent memory, characterized in that: It includes an input embedding layer that forms a closed-loop logic chain, a local neighborhood perception module, a pheromone field module, an agent update unit module, a group routing module, an emergent memory unit module, a feedback regulation module, a cross-group communication bus module, and an output aggregation module; The local neighborhood awareness module is used to construct a neighborhood graph for each agent node and generate neighborhood messages; The pheromone field module is used to update the global pheromone field matrix based on the release signal of the agent node and provides a feedback signal; The agent update unit module is used to integrate the agent state of the previous layer, neighborhood messages, pheromone feedback signals and cross-layer memory signals, and update the agent state through a gating mechanism. The group routing module is used to calculate the routing score based on the updated agent state and pheromone feedback signal, and generate the routing output; Emergent memory unit modules are used to write agent states, routing outputs, and pheromone feedback signals into the memory bank, and support on-demand reading to form cross-layer chain dependencies; The feedback adjustment module is used to calculate the adjustment factor based on pheromone field statistics, memory redundancy, and routing entropy, and to adjust the pheromone field and memory writing. The cross-group communication bus module is used to temporarily store and merge group communication results and field signal statistics in multi-group scenarios; The output aggregation module is used to fuse the final agent state, the emergent memory matrix, and the output of the cross-group coordination signal generation task.
2. A neural network architecture based on population routing and emergent memory according to claim 1, characterized in that: The local neighborhood perception module selects a K-order neighborhood set by calculating the cosine similarity between agent nodes, and generates a neighborhood weight matrix and neighborhood message aggregation results. The aggregation complexity is approximately O(NKd), and K is much smaller than the sequence length N.
3. A neural network architecture based on population routing and emergent memory according to claim 2, characterized in that: The formula for selecting the K-order neighborhood is as follows: Where cos is the cosine similarity, TopK returns the set of the K indices of the candidate set in descending order of a given score. It is the output of the upper-level proxy update unit; The formula for constructing the neighborhood weight matrix is as follows: ; The formula for constructing neighborhood message aggregation is as follows: Furthermore, the trainable mapping of the agent update unit channel is aligned to obtain the local neighborhood elimination of agent i. , , These are the parameters that the model learns during training. The expected message dimension for the agent update unit comes from the model structure constants.
4. A neural network architecture based on population routing and emergent memory according to claim 1, characterized in that: The pheromone update rule of the pheromone field module is as follows: ,in Let λ represent the pheromone field matrix of the Lth layer, where λ is the attenuation coefficient. The amount of new pheromone released by all agent nodes at layer L; When information is released, The calculation incorporates a dual coupling between the local neighborhood aggregation result and the agent's updated state. The calculation formula is as follows: , where η∈R + , is a positive real hyperparameter for controlling the release intensity, and "+" represents the set of positive real numbers; This represents the state of the layer above the i-th agent; The neighborhood messages output by the LNP at this layer; The release function maps the agent state and neighborhood messages together to the release signal.
5. A neural network architecture based on population routing and emergent memory according to claim 1, characterized in that: In the proxy update unit module, the proxy's input vector is first concatenated as follows: And introduce a gating mechanism The agent state is updated via gating, and the update formula is as follows: ; in, It is a trainable parameter matrix; σ is the bias term; σ is the Sigmoid function, which ensures that the gate value is between (0,1); This represents sequential multiplication; This represents the concatenation of vectors.
6. A neural network architecture based on population routing and emergent memory according to claim 5, characterized in that: In the group routing module at layer L, the input of each agent i is the updated state. pheromone feedback signals With neighborhood candidate set This is used to limit the candidate range for route selection; By defining a route scoring function R(i,j), the route strength between agent i and candidate neighbor j is characterized, and the formula is as follows: ,in, ∈R d×d It is a trainable parameter matrix; Indicates routing preference based on state similarity; γ represents the global pheromone-based signal; γ represents the global pheromone term weight. During route selection, agent i considers its candidate set N. K Neighbors are assigned routing weights in (i) The formula is The final routing output is .
7. A neural network architecture based on population routing and emergent memory according to claim 1, characterized in that: The inputs to the emergent memory unit module include the agent state. ∈R d Routing results With the global pheromone field Φ (L) ∈R n×d ; The emergent memory unit module uses a weighted write rule to write the communication results of the agent into the hierarchical memory. The formula is ,in ; Simultaneously, the emergent memory unit module supports on-demand reading, allowing agents to access M in subsequent layers. (L) Extracting historical chain memory This is the memory vector to be read, used by the next-level agent update unit module. Its reading mechanism is as follows: , where Attn() represents the attention-based retrieval function.
8. A neural network architecture based on population routing and emergent memory according to claim 1, characterized in that: The feedback adjustment module receives global pheromone field statistics. Redundancy of memory matrix Proxy routing entropy Input signal and generate global adjustment factor β (L) Global regulation factor β (L) Through formula The calculation yielded W. f b f For training parameters, β (L) ∈(0,1).
9. A neural network architecture based on population routing and emergent memory according to claim 1, characterized in that: The cross-group communication bus module is used to temporarily store and merge group sets G={G1,G2,…,Gm} and group communication results in multi-group scenarios. Field signal statistics Each of these groups Includes several agents, .
10. A neural network architecture based on population routing and emergent memory according to claim 1, characterized in that: The output aggregation module receives the agent state set at layer L=T. Emergent memory matrix With cross-group coordination signals The input is calculated using a formula to determine the output vector. The calculation formula is: ; in, It is the average of the final states of all agents. It is the average of the memory matrix. It is a cross-group signal, which is uniformly mapped to the output space by splicing the group average, memory average, and cross-group signal.