A multi-reference frame hierarchical mapping neural network system and attention mapping method

Abstract

The application discloses a multi-reference system hierarchical mapping neural network system and an attention mapping method, and belongs to the technical field of neural network attention mechanisms. The system comprises a multi-reference system attention module, which comprises a plurality of attention heads, each of which comprises a learnable reference system bias parameter, and an attention weight calculation circuit comprising an attention score modulation input end; a hierarchical abstract mapping module comprising three parallel mapping submodules, namely an instance mapping layer, a domain mapping layer and an abstract mapping layer, wherein the outputs of the submodules are spliced and then projected by an output projection matrix to replace a standard feedforward network. The corresponding method assigns the attention heads to a plurality of reference systems, and modulates the attention scores according to the reference system bias parameters in the attention weight calculation. The application enables different attention heads to use different relationship measurement methods, explicitly models the hierarchical abstract structure, and improves the performance of the neural network in multi-relationship type tasks.

Description

Technical Field

[0001] This invention relates to the field of neural network attention mechanism technology, specifically to a multi-reference frame hierarchical mapping neural network system, and an attention mapping method based on this system. Background Technology

[0002] In the multi-head attention mechanism of the standard Transformer architecture, all attention heads compute the attention weights between elements in the same feature space. Each head learns different attention patterns but shares the same "relationship metric"—namely, dot product similarity. This design assumes that the relationships between sequence elements can be uniformly measured by the inner product of feature vectors.

[0003] However, there are many different types of relationships between elements in real sequence data, requiring different measurement methods: Spatial relationships: the proximity and association of elements in a coordinate system or semantic space; Temporal relationship: The chronological order of elements on the timeline and their relationship to the periodic phase; Hierarchical relationship: The inclusion and belonging relationship between abstract concepts and concrete instances among elements; Directed relationship: The directional or logical order of influence between elements.

[0004] The four types of relationships mentioned above require different computational methods for measurement. However, the standard attention mechanism uses a uniform dot product similarity to handle all relationship types, making it impossible to employ different measurement methods for different relationship types. Furthermore, when deployed on a neural network accelerator (NPU), a uniform measurement method leads to different relationship types sharing the same hardware computation path, resulting in decreased utilization of the NPU's Single Instruction Multiple Data (SIMD) units and pipeline execution efficiency, and increased inference latency. Sequence data also contains hierarchical structure information—there are nested inclusion relationships between concepts, ranging from "macro-level → meso-level → micro-level," which the standard Transformer's linear feedforward network cannot effectively model.

[0005] In existing technologies, some methods introduce prior structural information by modifying attention bias terms, but they use a single bias pattern (such as distance decay or relative position encoding) and do not support the use of different types of biases in different attention heads. Regarding hierarchical modeling, tree-based long short-term memory networks and hyperbolic embedding methods suffer from low computational efficiency or gradient collapse when handling large-scale hierarchies. Existing technologies have also proposed methods to distinguish different types of attention heads—for example, distinguishing between sensory attention heads that encode the features of the elements themselves and relational attention heads that encode the relationships between elements. Relational attention heads calculate the relationships between elements through the inner product of learned feature maps. The limitations of this method are: (1) The relation type is not given a clear physical semantics. Attention heads are only distinguished by coarse-grained categories such as "sensory / association". It is impossible to use a measurement method that matches the properties of sequence relations with different properties such as space, time, hierarchy, and direction. (2) The relation measurement depends on the feature mapping obtained by learning. Its quality and interpretability depend on the distribution and scale of the training data. It cannot guarantee the applicability to relations with different properties. (3) There is no adaptive adjustment mechanism for the contribution weight of the relation measurement adopted by each attention head. The degree of participation of the relation measurement of each head in attention calculation cannot be automatically adjusted with the task. Summary of the Invention

[0006] The technical problem that this invention aims to solve is that existing attention mechanisms use a uniform metric to process all relation types, which cannot assign different relation metrics to different heads in multi-head attention. This results in a single hardware computation path, insufficient utilization of SIMD units, and a lack of modeling ability for hierarchical abstract structures in feedforward networks when deploying NPUs.

[0007] To address the aforementioned technical problems, this invention provides a multi-reference frame attention mapping method and a hierarchical abstract neural network system, which explicitly assigns attention heads to different reference frames, uses different relational metrics for each reference frame, and replaces the standard feedforward network with a hierarchical abstract mapping layer.

[0008] The technical solution adopted in this invention is as follows:

[0009] A multi-reference-frame hierarchical mapping neural network system includes: a multi-reference-frame attention module comprising attention heads assigned to multiple reference frames, each attention head including a learnable reference frame bias parameter b_h and a relation metric calculation circuit, wherein the relation metric calculation circuit calculates the relation metric value between element pairs using a corresponding relation metric function based on the type of the reference frame to which the attention head belongs; and a hierarchical abstract mapping module, the input of which is connected to the output of the multi-reference-frame attention module, comprising a three-layer parallel structure of an instance mapping layer, a domain mapping layer, and an abstract mapping layer. Optionally, the system further includes a configurable radix ALU unit array, comprising multiple configurable radix ALU units, each ALU unit including a radix configuration register and a carry-shift chain, for providing hardware acceleration support for multi-radix parallel computing for the multi-reference-frame attention module.

[0010] The corresponding multi-reference frame attention mapping method includes: assigning attention heads to multiple reference frames; setting learnable reference frame bias parameters for each attention head; modulating the attention score according to the reference frame bias parameters in the attention weight calculation; and fusing the outputs of all attention heads to generate a multi-reference frame enhanced sequence representation.

[0011] Furthermore, the multiple reference frames include a spatial reference frame, a temporal reference frame, a hierarchical reference frame, and a directed relational reference frame. Each reference frame uses a different relational metric function R_h(i,j)—the spatial reference frame measures the angular difference and distance, the temporal reference frame measures the periodic interval, the hierarchical reference frame measures the depth of the nearest common ancestor of the classification tree, and the directed relational reference frame measures the directional influence distance.

[0012] Furthermore, each attention head carries a learnable scalar parameter b_h as a reference bias, which is used as a modulation term attn_weight(h,i,j) = softmax(Q_i·K_j^T / √d + sigmoid(b_h)·R_h(i,j)) in the attention weight calculation; b_h is initialized to 0 and is automatically adjusted through backpropagation during training. When it approaches 1, the bias term plays a major role, and when it approaches 0, it degenerates into a standard dot product attention.

[0013] Furthermore, the hierarchical abstract mapping layer is implemented through three parallel linear projections: the instance mapping layer outputs d_ff dimension, the neighborhood mapping layer outputs d_ff / 2 dimension, and the abstract mapping layer outputs d_ff / 2 dimension. The three output vectors are concatenated and then fused back to the original feature dimension through linear projection W_o.

[0014] Furthermore, the relationship measurement of the hierarchical structure is implemented by deterministic polar coordinate slicing or binary lifting circuit to calculate the depth of the nearest common ancestor. For small-scale levels (N less than the first preset threshold), polar coordinate slicing is used, while for large-scale levels (N greater than the second preset threshold), a hardware circuit scheme of binary lifting circuit combined with block memory table lookup is used. When the level size is between the first preset threshold and the second preset threshold, either of the above methods can be selected.

[0015] Beneficial effects: (1) Different attention points use different relation measurement methods, enabling the model to capture multiple types of relation information at the same time; (2) The reference frame bias parameters can be learned, and the model automatically determines the utilization degree of each reference frame; (3) Hierarchical abstract mapping explicitly models the three-layer structure of "instance → domain → abstraction", improving the ability to model hierarchical relationships; (4) The deterministic slicing method of small-scale hierarchy avoids the gradient collapse problem. Attached Figure Description

[0016] Figure 1 The diagram below illustrates the multi-reference frame attention mapping of the present invention (110 is the input sequence representation, 120 is the QKV projection unit, 130 is the attention head allocator, 140 is the reference frame bias register b_h, and 150 is the output merging unit). It shows the attention head being allocated to four reference frames: spatial, temporal, hierarchical, and directed, as well as the modulation process of the reference frame bias parameter b_h on the attention score.

[0017] Figure 2 The diagram below shows the structure of the hierarchical abstract mapping layer of this invention (210 is the input, 220 is the instance mapping layer, 230 is the domain mapping layer, 240 is the abstract mapping layer, 250 is the splicing unit, and 260 is the output projection unit W_o). It illustrates the structure of the three parallel mapping layers of instance mapping layer, domain mapping layer and abstract mapping layer, which are spliced ​​and output through linear projection W_o.

[0018] Figure 3 The diagram shows the overall architecture of the neural network system of the present invention, illustrating the positions of the multi-reference attention module (310) and the hierarchical abstract mapping module (320) in the Transformer architecture, as well as the optional configurable radix ALU unit array (100) – which, when a hardware mapping scheme is adopted, serves as a hardware base to provide multi-radix parallel computing support for four reference frames. Detailed Implementation

[0019] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0020] Example 1: Attention Allocation in Multiple Reference Frames

[0021] This invention assigns multiple attention heads of a neural network to different reference frames. In a typical configuration, four reference frames correspond to spatial relation metrics, temporal relation metrics, hierarchical relation metrics, and directed relation metrics, respectively, and eight attention heads are assigned to each reference frame in pairs.

[0022] Each attention head h carries a learnable scalar parameter b_h, called the reference frame bias. This bias is used as a modulation term in the attention weight calculation:

[0023] attn_weight(h,i,j) = softmax(Q_i · K_j^T / √d + sigmoid(b_h) · R_h(i,j))

[0024] Where R_h(i,j) is the relational measure of elements i and j in the reference frame of the h-th attention head.

[0025] The calculation method for R_h(i,j) varies depending on the reference frame:

[0026] (1) Spatial reference frame: R(i,j) = cos(θ_ij) / (1 + d_ij), where θ_ij is the angular difference between elements i and j in the embedded space, and d_ij is the Euclidean distance.

[0027] (2) Time reference frame: R(i,j) = sin(2π · Δt / T), where Δt is the timestamp difference and T is the reference period (configurable).

[0028] (3) Hierarchical reference system: R(i,j) = depth(LCA(i,j)), where depth(LCA(i,j)) is the depth of the nearest common ancestor of the classification tree of elements i and j. The closer the hierarchy (the greater the depth of the nearest common ancestor), the greater the relational measure.

[0029] (4) Directed relation reference frame: R(i,j) = D_ij, where D_ij is the directional distance based on the directional influence relationship between elements. The directional influence relationship is determined according to the relative order of elements in the sequence or the causal direction.

[0030] The parameters in the above formulas can be configured according to task requirements. In the spatial reference frame, the distance metric d_ij can be replaced with Euclidean distance, Manhattan distance, or cosine distance, and the calculation dimension of θ_ij can be limited to the first k principal components of the embedded space (k is a configurable parameter). In the temporal reference frame, the value of the reference period T is determined by the time scale of the task—in fine-grained sequence tasks (such as sensor sampling), T takes a fixed value at the millisecond level, while in coarse-grained sequence tasks (such as document analysis), T can be set as a learnable parameter or provided by an external calendar system (such as T=24h, T=7d, T=12m). In the hardware-accelerated implementation, the time difference Δt in the relational metric R(i,j)=sin(2π·Δt / T) of the temporal reference frame can be calculated by a time-aware coprocessor hardware acceleration. The time-aware coprocessor includes a nested period counter array and a period engine, which outputs the periodic phase difference between any two event time points in a single cycle in combinational logic. In a hierarchical reference system, the depth of the nearest common ancestor can be normalized to the [0,1] interval according to the hierarchical path length, making the relationship metrics between hierarchical structures of different scales comparable. The directional distance D_ij of the directed relation reference system can be either absolute distance or squared distance; the latter is more suitable for scenarios where the intensity of directional influence needs to be amplified. The relationship metrics of the above reference systems are calculated directly from the input data through explicit formulas, rather than being learned.

[0031] The value of b_h is initialized to 0 before training, at which point sigmoid(0) = 0.5, and the initial weight of the bias term is moderate. During training, b_h is automatically adjusted through backpropagation: if the information of a certain reference frame is beneficial to the task, sigmoid(b_h) approaches 1, and the bias term plays a major role; if the information of the reference frame is irrelevant or interfering, sigmoid(b_h) approaches 0, and the attention degenerates into standard dot product attention.

[0032] Example 2: Hierarchical Abstraction Mapping Layer

[0033] The hierarchical abstraction mapping layer replaces the feedforward network layer in the standard Transformer, extending the linear transformation into a three-layer parallel mapping structure that captures features at different abstraction levels.

[0034] The definition of a three-layer mapping is as follows:

[0035] Instance Mapping Layer: Maps input features to an instance-level representation space, preserving fine-grained features of specific elements. The output dimension of this layer is the same as the hidden layer dimension of a standard FFN.

[0036] Domain Mapping Layer: Maps input features to a domain constraint representation space, capturing common features of the domain to which the current element belongs. The output dimension of this layer is half that of the instance mapping layer.

[0037] Abstract Mapping Layer: Maps input features to a cross-domain general structural representation space, capturing domain-independent structural patterns. The output dimension of this layer is half that of the Instance Mapping Layer.

[0038] Three-layer mapping is achieved through linear projection:

[0039] specific = GELU(W_s × x)

[0040] domain = GELU(W_d × x)

[0041] abstract = GELU(W_a × x)

[0042] output = W_o × concat(specific, domain, abstract)

[0043] Where x is the input feature vector, W_s, W_d, and W_a are the projection matrices of the instance layer, domain layer, and abstract layer, respectively, the total output dimension is the sum of the three vector dimensions of specific, domain, and abstract, W_o projects the concatenated vector back to the original feature dimension, and GELU is the activation function.

[0044] In a typical Transformer configuration, if the hidden layer dimension is d_ff, then the instance mapping layer outputs d_ff dimension, the domain mapping layer outputs d_ff / 2 dimension, and the abstract mapping layer outputs d_ff / 2 dimension. After concatenation, the result is 2×d_ff dimension, which is then projected back to d_ff dimension via W_o.

[0045] Example 3: Small-scale hierarchical deterministic fragmentation

[0046] For small-scale hierarchical structures with fewer than a first preset threshold (typically 100 nodes), this invention uses a deterministic subtree angle slicing method to replace gradient-optimized embedding methods (such as Poincaré embedding).

[0047] The specific method is as follows: Taking the root node of the hierarchical structure as the origin of polar coordinates, the subtree is uniformly divided into k segments along the angular direction (k is a preset value, typically 6 to 12). Each segment covers an angle interval of 360 / k. The position of a node in polar coordinates is determined by two parameters: the angle θ is determined by the segment where the node is located (the center angle of the segment), and the radius r is determined by the depth of the node in the hierarchical tree (the depth value is normalized to the range of 0 to 1).

[0048] The relationship metric between two nodes in a hierarchical reference frame is calculated as follows:

[0049] R(i,j) = -[|r_i - r_j| + r_i · |sin(θ_i - θ_j)|]

[0050] The first metric measures the depth difference, and the second metric measures the angle difference. The smaller the combined value of the two, the closer the hierarchical distance.

[0051] Example 3A: Hardware circuit implementation for calculating the depth of the nearest common ancestor

[0052] For large-scale hierarchical structures (where the number of nodes N is greater than a second preset threshold, typically 1000 nodes), the hardware implementation of the Least Common Ancestor (LCA) depth calculation adopts a binary lifting preprocessing plus block memory lookup scheme. In the preprocessing stage, the encoding circuit traverses the hierarchy tree to generate a 2^k-level ancestor index table for each node, stored in on-chip block memory—each node stores a ceil(log2(N))-level ancestor index, with each index encoded in ceil(log2(N)) bits, for a total storage of N×ceil(log2(N))×ceil(log2(N)) bits. In the lookup stage, the lookup control circuit reads and compares the ancestor indices of two nodes from the highest to the lowest level, outputting the Least Common Ancestor depth value. The block memory uses synchronous reads, with the comparison proceeding in a pipelined manner. The lookup latency is proportional to the number of lifting levels, approximately 2×ceil(log2(N)) clock cycles. The binary boosting circuit scheme complements the deterministic polar coordinate slicing method of Example 3—the binary boosting circuit is used for large-scale levels (N greater than the second preset threshold), while the polar coordinate slicing method is used for small-scale levels (N less than the first preset threshold); any method can be used for levels between the two preset thresholds. In a system with multiple level reference frames and attention heads, N does not exceed the total number of classification tree nodes supported by the attention head, with a typical value of N≤512. The storage overhead does not exceed 512×9×9=41,472 bits (approximately 5KB), which can be efficiently deployed on FPGAs that support block memory. The above storage and deployment metrics have been experimentally verified: on a low-cost FPGA with approximately 20,000 logic cells, the 41,472-bit ancestor index table occupies 3 blocks of block memory (approximately 7% of the device's block memory resources), the query and pipeline control logic occupies approximately 1% of the logic resources, and the overall clock frequency exceeds 100MHz without timing violations, verifying the feasibility of the above storage requirements and pipelined query method on low-cost FPGAs.

[0053] Example 4: Multi-reference frame integrated mode selection circuit

[0054] The multi-reference frame integration mode selection circuit achieves hardware-level switching between alternative and supplementary integration modes through a data path multiplexer. Integration modes include:

[0055] (1) Alternative mode: Multi-reference attention module directly replaces standard multi-head self-attention layer, and hierarchical abstract mapping layer replaces standard feedforward network layer;

[0056] (2) Additional mode: The multi-reference attention module is inserted as an additional layer between the standard Transformer layers, and the hierarchical abstract mapping layer and the standard feedforward network layer are output in parallel and then fused.

[0057] In the alternative mode, each layer of the Transformer encoder contains a multi-reference attention sublayer and a hierarchical abstract mapping sublayer, maintaining compatibility with the standard Transformer architecture. The training process is consistent with the standard Transformer, with only two additional sets of parameters added: b_h and the three-layer mapping projection matrix.

[0058] The two integration modes are switched via a data path selection circuit. The selection circuit contains two sets of multiplexers: (i) an attention path selector, which in alternative mode directly connects the output of the multi-reference attention module to the input of the hierarchical abstraction mapping layer or the standard feedforward network; in additional mode, it merges the output of the standard attention module with the output of the multi-reference attention module in a summing unit before passing it to the next layer. (ii) a feedforward path selector, which in alternative mode uses the output of the hierarchical abstraction mapping layer as the output of the encoder layer; in additional mode, it merges the output of the hierarchical abstraction mapping layer with the output of the standard feedforward network in a splicing unit and outputs it via linear projection. The mode selection signal is driven by the integrated configuration register, remains stable after being written during model initialization, and does not participate in dynamic switching during runtime. The additional logic resource consumption of the selection circuit is within approximately 50 equivalent logic gates, without changing the bit width and clock frequency of the main data path.

[0059] Example 5: Hardware Mapping of Neural Network Accelerators

[0060] This embodiment provides a computational pipeline scheme for deploying multi-reference frame attention mapping and hierarchical abstraction mapping layers to a neural network accelerator (NPU). The multi-reference frame attention module (310) and the hierarchical abstraction mapping module (320) serve as two-level computing units of the accelerator, and are executed in a pipelined manner through data scheduling of on-chip memory.

[0061] The input feature sequence is loaded into the accelerator's input buffer (110) as batch tensors with shape (B, L, d_model), where B is the batch size, L is the sequence length, and d_model is the embedding dimension. The accelerator's matrix multiplication unit performs QKV projection computation (120), and the three projection matrices W_q, W_k, and W_v generate query tensor Q, key tensor K, and value tensor V with shape (B, L, d_k), respectively. d_k is calculated by dividing d_model by the number of attention heads.

[0062] The attention head allocator distributes eight attention heads into four reference frames (130) according to a preset grouping, with each group containing two attention heads. The reference frame bias parameter b_h (140) for each attention head h is stored in the accelerator's scalar parameter register set. Attention score calculation:

[0063] The first step is to calculate the dot product of Q and K^T using the matrix multiplication unit. The resulting tensor has the shape (B, h_num, L, L).

[0064] The second step is that the vector processing unit performs bias modulation on the attention head of each reference frame. After reading the b_h value from the scalar parameter register, it converts it into modulation coefficients through the sigmoid function and applies the bias term sigmoid(b_h)·R_h(i,j) to the corresponding position of the fractional tensor.

[0065] The third step is to perform exponential normalization along the last dimension of the attention fraction tensor using the softmax normalization unit.

[0066] The fourth step involves the weighted summation unit summing the normalized weights and the value tensor V to output the multi-reference frame enhanced attention result.

[0067] The hierarchical abstract mapping module (320) receives the output of the multi-reference attention module. The instance mapping layer (220), the domain mapping layer (230), and the abstract mapping layer (240) synchronously perform GELU-activated linear projections in three parallel computing channels of the accelerator. The output dimension of the instance mapping layer is d_ff, and the output dimensions of the domain mapping layer and the abstract mapping layer are each d_ff / 2. The three output tensors are merged into a 2×d_ff dimensional tensor in the splicing unit (250), and then projected back to the d_model dimension by the output projection unit (260) through the matrix W_o, completing the full computation of one encoder layer.

[0068] Example 6: Typical Configuration Parameters

[0069] When deploying multi-reference attention mapping and hierarchical abstraction mapping layers to a 12-layer Transformer encoder, typical configuration parameters for each module include embedding dimension, number of attention heads, number of reference frames, and adjustable range of each item. The specific configuration parameters are as follows.

[0070] Typical configuration parameters for each module are as follows: the encoder has 12 layers, each containing a multi-reference frame attention module and a hierarchical abstract mapping module; the embedding dimension d_model is 512, serving as the common dimension for input and output features; the total number of attention heads is 8, with 2 attention heads assigned to each reference frame (i.e., 2 heads / reference frame multiplied by 4 reference frames); the dimension d_k of each head is 64, equal to d_model divided by the number of attention heads; the feedforward hidden layer dimension d_ff is 2048, serving as the output dimension of the instance mapping layer; the domain layer and abstract layer dimensions are 1024, equal to half of d_ff; the initial reference frame bias value b_h is 0, corresponding to an initial bias of 0.5 after passing through the sigmoid function; the number of reference frames is 4, corresponding to the four types of relationships: spatial, temporal, hierarchical, and directed; the batch size B is 16 to 64, configurable based on the accelerator's memory capacity; the maximum sequence length L is 512, configurable from 128 to 2048 during training.

[0071] The parameters added in the above configuration come from two parts: (1) four sets of reference frame biases b_h, four scalars per layer, the increase is negligible; (2) three sets of projection matrices of the hierarchical abstract mapping layer, replacing the two sets of projection matrices (W1, W2) of the standard feedforward network, the increase is about 2×d_ff×d_model (the added W_o projects the three-layer concatenation result back to the original dimension) - the projection parameters of the standard feedforward network are 2×d_model×d_ff, and the hierarchical abstract mapping layer is d_model×d_ff + 2×(d_ff / 2)×d_model + 2×d_ff×d_model = 4×d_model×d_ff, the total number of projection parameters is about twice that of the standard feedforward network. This increment is the structural design choice of the hierarchical abstract mapping layer to exchange more parameters for the ability of three layers of parallel feature extraction, and does not belong to the parameter overhead of the multi-reference frame attention module itself.

[0072] The above parameters can be adjusted before training according to task requirements: the total number of attention heads can be between 4 and 16 (the number of heads must be an integer multiple of the number of reference frames to ensure that each reference frame is allocated the same number of attention heads), the number of reference frames can be between 2 and 8, and the number of encoder layers can be between 6 and 24. The above adjustments do not change the core calculation method of multi-reference frame attention mapping.

[0073] Example 7: Application of Sequence Modeling with Multiple Relation Types

[0074] This embodiment provides a scheme for applying multi-reference frame attention mapping to sequence modeling with multiple relationship types. In modeling relationships between sequence elements, the interaction relationships between elements include various types: proximity of elements in semantic space (spatial relationship), sequential interval of elements on the time axis (temporal relationship), hierarchical affiliation between concepts (hierarchical relationship), and chain of influence between elements (directed relationship). Standard attention models use a uniform dot product similarity metric to measure relationships between all elements, failing to distinguish the different weights of different types of relationships in sequence modeling.

[0075] This embodiment applies multi-reference frame attention to sequence encoding: the spatial reference frame measures the distance between elements in the semantic space, the temporal reference frame measures the periodic interval between elements, the hierarchical reference frame measures the belonging depth of a concept in the hierarchical tree, and the directed relation reference frame measures the causal influence of preceding events on subsequent events. The advantage of the multi-reference frame attention mechanism lies in extending a single dot product similarity metric to parallel computation of multiple relation metrics—different attention heads specialize in a single relation type, and each head automatically adjusts its relation metric's contribution weight in attention weight calculation through learnable reference frame bias parameters—achieving richer relation modeling capabilities between sequence elements compared to the unified metric method of standard multi-head attention. The hierarchical abstract mapping layer, through a three-layer parallel structure of instance mapping layer, domain mapping layer, and abstract mapping layer, extends the feedforward network from a single-layer linear transformation to a three-layer hierarchical feature extraction, capturing specific instance features, domain-shared features, and cross-domain general structural patterns at different abstraction levels, achieving stronger hierarchical feature representation capabilities under equal parameter constraints.

[0076] The multi-reference frame attention mechanism of this invention adds only a small number of learnable reference frame bias parameters and a three-layer parallel structure compared to the standard multi-head attention mechanism. The overall parameter increment is limited, making it suitable for deployment on edge devices or older hardware with limited computing resources.

[0077] Example 8: Optional hardware deployment on a configurable base number ALU cell array

[0078] Optionally, when the multi-reference frame attention module of the present invention needs to be deployed on a hardware accelerator, the following hardware mapping scheme can be adopted. The four reference frames of the multi-reference frame attention module—spatial reference frame, temporal reference frame, hierarchical reference frame, and directed relational reference frame—are mapped to four groups of ALU units in a configurable radix ALU unit array. Each group of ALU units contains at least two ALU units, with two attention heads allocated to each reference frame. The radix configuration register of each ALU unit is independently set: the ALU units of the spatial reference frame group are configured with floating-point radix for direction angle difference and distance calculation; the ALU units of the temporal reference frame group are configured with fixed-point decimal radix, working with CORDIC combinational logic circuits to complete trigonometric function operations in time offset and phase difference calculations; the ALU units of the hierarchical reference frame group are configured with logarithmic radix for nearest common ancestor depth calculation; and the ALU units of the directed relational reference frame group are configured with ternary radix for direction and distance calculation. The front-end decomposer decomposes the input data into four mixed radix representations according to the configured radix of each group, and sends them to the four groups of ALU units respectively. Each ALU unit calculates its relational measure in parallel within the same clock cycle. The calculation results are then merged by a back-end reassembler into a multi-reference-frame fused attention output. The hardware mapping scheme described above is not a necessary implementation of the multi-reference-frame attention mechanism of this invention—the four relational measure functions of the multi-reference-frame attention module can also be implemented in software or firmware on a general matrix operation unit (such as the matrix multiplication unit of a standard GPU or NPU).

[0079] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-reference frame hierarchical mapping neural network system, characterized in that, include: The multi-reference frame attention module includes multiple attention heads, which are divided into multiple groups. Each group is mapped to a preset reference frame. Each attention head includes a learnable reference frame bias parameter b_h and a relation metric calculation circuit. The relation metric calculation circuit calculates the relation metric value between element pairs using the corresponding relation metric function R_h(i,j) according to the type of the reference frame to which the attention head belongs. The preset reference frame includes at least two of the following: The spatial reference frame has a relation metric function R(i,j) = cos(θ_ij) / (1+d_ij), where θ_ij is the angular difference of the elements in the embedding space and d_ij is the distance between the elements; The time reference frame has a relational metric function R(i,j) = sin(2π·Δt / T), where Δt is the difference in element timestamps and T is the configurable reference period; The hierarchical reference system has a relational metric function R(i,j) = depth(LCA(i,j)), where depth(LCA(i,j)) is the depth of the lowest common ancestor of the element in the classification tree; A directed relational reference system has a relational metric function R(i,j) = D_ij, where D_ij is the directional distance between elements; The output of the relation metric calculation circuit is connected to the attention score modulation input. The attention weight calculation circuit calculates the attention weight according to the formula attn_weight(h,i,j) = softmax(Q_i · K_j^T / √d + sigmoid(b_h) · R_h(i,j)), where Q_i and K_j are the query vector and key vector, respectively, d is the attention dimension, and b_h is the reference frame bias parameter of the h-th attention head. The hierarchical abstract mapping module, whose input is connected to the output of the multi-reference attention module, includes three parallel mapping sub-modules: an instance mapping layer sub-module, a domain mapping layer sub-module, and an abstract mapping layer sub-module. Each sub-module contains independent linear projection matrices W_s, W_d, and W_a. The output of the three parallel mapping sub-modules is connected to the stitching layer, and the output of the stitching layer is connected to the output of the hierarchical abstract mapping module via the output projection matrix W_o, replacing the standard feedforward network layer.

2. The system according to claim 1, characterized in that, It also includes a configurable radix ALU unit array, wherein multiple attention heads of the multi-reference frame attention module are respectively mapped to multiple groups of ALU units in the configurable radix ALU unit array, and each group of ALU units independently calculates the relation metric of a reference frame; the front end of the configurable radix ALU unit array includes a hybrid radix data decomposition and recombination unit, which decomposes the external binary input data into multiple hybrid radix representations according to the configured radix of each ALU unit.

3. The system according to claim 1, characterized in that, The system is integrated with the Transformer architecture, where the multi-reference attention module replaces the standard multi-head self-attention layer, and the hierarchical abstract mapping module replaces the standard feedforward network layer; or the multi-reference attention module and the hierarchical abstract mapping module are embedded as additional layers in the existing Transformer architecture.

4. The system according to claim 1, characterized in that, The time difference Δt in the relational metric of the time reference system is calculated by a time-aware coprocessor hardware acceleration. The time-aware coprocessor includes a nested period counter array and a period engine. The period engine outputs the periodic phase difference between any two time points within a single clock cycle through combinational logic.

5. A multi-reference frame attention mapping method, applied to the system described in claim 1, characterized in that, Executed on a computing device including a neural network accelerator (NPU), the NPU including a matrix multiplication unit, a vector processing unit, and a softmax normalization unit, the method includes: The attention head of the neural network is assigned to multiple predefined reference frames. Each reference frame defines a relational metric between elements, which is calculated directly from the input data by an explicit formula rather than being learned. A learnable reference frame bias parameter is set for each attention head, and the bias parameter controls the sensitivity of the corresponding attention head to different reference frames; In the calculation of attention weights, the attention score is modulated according to the reference frame bias parameter of each attention head: attn_weight(h,i,j) = softmax(Q_i · K_j^T / √d + sigmoid(b_h) · R_h(i,j)) Where R_h(i,j) is the relational measure between element i and element j in the reference frame of the h-th attention head, and b_h is the reference frame bias parameter of the h-th attention head; By fusing the outputs of all attention heads, a multi-reference frame enhanced sequence representation is generated.

6. The method according to claim 5, characterized in that, The plurality of reference frames includes at least two of the following: Spatial reference frame: a measure of the positional distance and orientation vectors of elements in physical or semantic space; Time reference frame: based on the chronological order of elements on the time axis and the phase difference measurement relationship; Hierarchical reference system: measures relationships based on the category level and nesting / containment relationships of elements; Directed Relationship Reference Frame: Measures relationships based on the directional influence between elements.

7. The method according to claim 5, characterized in that, It also includes a hierarchical abstraction mapping layer, which replaces the feedforward network layer of the neural network and contains three layers of parallel mapping: Specific instance mapping: Maps the current element to an instance-level feature representation, preserving the element's fine-grained features; Domain constraint mapping: Maps the current element to the structural constraint space of its domain, capturing common features within the domain; Abstract Structure Mapping: Maps the current element to a cross-domain common structure pattern space, capturing shared structural features across domains; The outputs of the three-layer mapping are concatenated along the feature dimension and then output via linear projection.

8. The method according to claim 7, characterized in that, The three-layer parallel mapping is implemented through three independent linear projection layers: specific = GELU(W_s × x) domain = GELU(W_d × x) abstract = GELU(W_a × x) output = W_o × concat(specific, domain, abstract) Where x is the input feature vector, W_s, W_d, and W_a are the projection matrices of the instance layer, the domain layer, and the abstract layer, respectively, W_o is the output projection matrix, and GELU is the activation function.

9. The method according to claim 6, characterized in that, The relational measure of the spatial reference system is based on the Euclidean distance and direction cosine of the element in the embedded space; the relational measure of the temporal reference system is based on the temporal offset calculation of the element's timestamp. The relational measure of the hierarchical reference system is based on the depth of the nearest common ancestor and the hierarchical distance of the elements in the classification tree; the relational measure of the directed relational reference system is based on the directional distance of the directional influence relationship between elements, which is determined according to the relative order or causal direction of the elements in the sequence.

10. The method according to claim 9, characterized in that, The relational metric function of the spatial reference system is R(i,j) = cos(θ_ij) / (1 + d_ij), where θ_ij is the angular difference of the element in the embedding space, and d_ij is the Euclidean distance; the relational metric function of the temporal reference system is R(i,j) = sin(2π·Δt / T), where Δt is the timestamp difference, and T is the configurable reference period; the relational metric function of the hierarchical reference system is R(i,j) = depth(LCA(i,j)), where depth(LCA(i,j)) is the depth of the lowest common ancestor of the element in the classification tree; the relational metric function of the directed relational reference system is R(i,j) = D_ij, where D_ij is the directional distance; the reference system bias parameter b_h is initialized to 0 before training, making sigmoid(0) = 0.5; During training, the sigmoid(b_h) is automatically adjusted through backpropagation. When a reference frame is beneficial to the task, the sigmoid(b_h) approaches 1, and when the reference frame information is irrelevant or interfering, the sigmoid(b_h) approaches 0.

Images