A neural network training system and sequence relation encoding method based on directionality encoding

Abstract

The application discloses a neural network training system and a sequence relationship coding method based on direction coding, and belongs to the technical field of neural network sequence modeling. The system comprises a direction coding module, a directional relationship graph storage circuit and a base vector coding circuit, and outputs explicit coding values of a forward base vector +1 and a reverse base vector -1; an attention enhancement module, which superimposes a direction distance on an attention score; a direction constraint module, which calculates a direction constraint loss according to a known directional relationship pair; and a feature fusion module, which fuses the direction base vector and embedded features. The corresponding method encodes the directional relationship into the direction base vector, introduces a directional bias in the attention weight calculation, and strengthens the utilization of directional information through the directional constraint loss. The application enables the neural network to explicitly distinguish the directional relationship between sequence elements, and improves the modeling accuracy on tasks with a directional relationship structure.

Description

Technical Field

[0001] This invention relates to the field of neural network sequence modeling technology, specifically to a neural network training system based on directional encoding, and a sequence relation encoding method based on this system. Background Technology

[0002] Existing neural network sequence modeling methods (especially the Transformer architecture) primarily rely on attention mechanisms to calculate the similarity between element features when handling relationships between elements. The standard attention weight is calculated as softmax(QK^T / √d), which is entirely based on the dot product similarity of feature vectors and does not distinguish the direction of the relationship.

[0003] However, the relationships between elements in sequence data are often directional. Taking natural language as an example, in the statement "Because A, therefore B," there is a clear unidirectional influence relationship between A and B. However, standard attention mechanisms use the same feature similarity metric when calculating the attention weights for A→B and B→A, failing to distinguish the direction. In time series, the direction of influence of preceding events on subsequent events is semantically different from the direction of tracing back from subsequent events to preceding events, and standard time series models (such as RNNs and Transformers) do not explicitly encode this directional difference.

[0004] In existing technologies, some works introduce relative position encoding (such as RoPE) or causal attention masks. While masking is used to model sequence direction information, Transformer schemes for directed graph data also propose encoding for graph topology. However, these methods have the following drawbacks: (1) Causal attention masking simply masks future information and cannot encode the strength of directional influence; (2) Relative position encoding encodes position offset rather than relational direction, and there is no direct correspondence between the directional influence relationship between two elements and the position offset; (3) Existing methods do not support encoding both positive influence and reverse tracing directional information in the same model at the same time; (4) The encoding of Transformer schemes for directed graphs is designed for the adjacency structure of nodes and edges in graph topology, and is oriented towards graph data form. It cannot be directly applied to the directional relationship between element pairs in sequence data. The directional influence relationship between sequence elements (such as causal relationship, temporal transmission relationship) is a different data form from the graph topology edge and needs to be explicitly encoded in fusion with the sequence feature representation; (5) Existing directional encoding schemes do not include directional constraint supervision mechanism, and cannot constrain the consistency between the model prediction result and the known directional relationship during training. Directional information only participates in forward calculation as input feature, and its utilization cannot be explicitly strengthened during training. Summary of the Invention

[0005] The technical problem this invention aims to solve is that existing neural network attention mechanisms cannot distinguish the directionality of relationships between elements, cannot explicitly encode the positive influence direction and the backward tracing direction, and do not support learnable modulation of directional information. When deployed on a neural network accelerator (NPU), the lack of directional information causes the causal direction and the backtracking direction to share the same hardware computation path, making it impossible to use directional signals to achieve parallel optimization of the computation pipeline.

[0006] To address the aforementioned technical problems, this invention provides a sequence relation encoding method based on directional basis vectors. The method encodes directed relations into coordinate representations composed of forward and backward basis vectors, integrates them into the feature representation of a neural network, and enhances the utilization of directional information during training through a directional constraint loss function.

[0007] The technical solution adopted in this invention is as follows:

[0008] A training system for a sequence relation encoding neural network based on directional basis vectors includes: a directional encoding module, whose input receives an input sequence and outputs directional basis vectors, wherein the directional basis vectors are directly determined by directed relations rather than learned, and the projection values ​​of the elements on the direction coordinate axes are taken from the three-value set {+1,-1,0}; an attention enhancement module, whose input is connected to the output of the directional encoding module, and which superimposes the directional bias term α_h·D_ij onto the attention score; a directional constraint module, whose input is connected to the model prediction output, and which calculates the directional constraint loss L_dir based on the known directed relations; and a feature fusion module, whose first input is connected to the feature vector output of the embedding layer, and whose second input is connected to the output of the directional encoding module. Optionally, the system further includes a non-binary radix arithmetic unit, which uses positive integers other than powers of 2 as the radix, and the bit selection logic of the basis vector encoding circuit and the non-binary radix arithmetic unit is multiplexed using bit lines.

[0009] The corresponding sequence relation encoding method includes: obtaining directed relation information between sequence elements; encoding the directed relation information into forward basis vectors and backward basis vectors; fusing the directional basis vectors with the feature representation to generate a directionally enhanced embedding vector; and inputting the directionally enhanced embedding vector into a neural network.

[0010] Furthermore, the projection values ​​of the directional basis vectors are taken from the three-value set {+1,-1,0}. The directional distance is defined as the absolute value of the difference between the projection values ​​of two elements on the directional coordinate axis. A distance of 2 indicates that there is a directional influence and the direction is clear, while a distance of 0 indicates that there is no directional relationship or the directions are parallel.

[0011] Furthermore, the attention enhancement module introduces a directional bias term α_h·D_ij in the attention weight calculation, where α_h is the learnable scaling parameter of the h-th attention head and D_ij is the directional distance between elements i and j; α_h is automatically adjusted through backpropagation during training, and its value reflects the intensity of the utilization of directional information in attention calculation.

[0012] Furthermore, the directional constraint loss function L_total = L_base + λ·L_dir, where L_dir is the directional constraint term and λ is the balance coefficient; L_dir is calculated as the sum of the negative log probabilities of known directed relation pairs (i,j), forcing the model to use the encoded directional information for prediction.

[0013] Furthermore, the directed relation graph is automatically constructed from the input sequence through trigger word matching or temporal inference and stored in the form of an adjacency matrix. The matrix elements take values ​​of {+1,-1,0}, which represent forward, reverse, and no relation, respectively.

[0014] Beneficial effects: (1) Explicitly encodes the directional relationship between sequence elements, enabling the model to model in both the forward (source → target) and reverse (target → source) directions respectively; (2) The directional bias is learnable, and the model automatically adjusts the intensity of directional information utilization during training; (3) The directional constraint loss ensures that the prediction results are consistent with the known directional relationship, reducing directional reasoning errors; (4) The directional relationship graph is automatically constructed from the original sequence data without the need for external knowledge base support. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the directional basis vector encoding of the present invention (110 is the input sequence, 120 is the direction coordinate axis, 130 is the forward basis vector, 140 is the reverse basis vector, 135 is the neutral basis vector, and 150 is the direction distance calculation), which shows the process of mapping the directional relationship between sequence elements to the projection values ​​of {+1,-1,0} on the direction coordinate axis.

[0016] Figure 2 The diagram below shows the structure of the attention enhancement module of this invention (210 is the query tensor Q, 220 is the key tensor K, 230 is the value tensor V, 240 is the directional bias attention calculation unit, and 250 is the directional enhancement attention output). It illustrates the process of incorporating the directional basis vector into the attention weight calculation through the directional bias term α_h·D_ij. The structure of the independent α_h parameters of different attention heads in multi-head attention is described in the following text

[0029] .

[0017] Figure 3This is a schematic diagram of the calculation process of the directional constraint loss function of the present invention (310 is the training data, 320 is the directed relation graph construction unit, 330 is the adjacency matrix storage unit, 340 is the directional constraint loss calculation unit, 350 is the total loss calculation unit, and 360 is the backpropagation unit). It shows the entire process from constructing a directed relation graph from the training data, generating an adjacency matrix, calculating the directional constraint term L_dir and combining it with the basic loss L_base to form the total loss L_total, and finally updating the parameters through backpropagation.

[0018] Figure 4 The diagram shows the overall architecture of the neural network training system of this invention (100 is an optional non-binary radix arithmetic unit, 410 is the input sequence, 420 is the embedding layer, 430 is the directional coding module, 440 is the feature fusion module, 450 is the attention enhancement module, and 460 is the directional constraint module). It illustrates the main data path and loss calculation loop of each module of the system. Among them, 100 is an optional hardware implementation base, corresponding to the deployment mode of the non-binary radix arithmetic unit. The directional coding mechanism of this invention does not depend on this hardware base. Detailed Implementation

[0019] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0020] Example 1: Directional basis vector encoding

[0021] The core idea of ​​directional basis vector encoding is to map the directed relationships between sequence elements into projection values ​​on a one-dimensional directional coordinate axis. Specifically, for element pairs with a directed relationship i→j (meaning that element i has a directional influence on element j), the projection value of element i on the directional coordinate axis is set to +1 (forward basis vector), and the projection value of element j is set to -1 (backward basis vector). Elements without a directional relationship have a projection value of 0 on the coordinate axis.

[0022] Directional distance is defined as the absolute value of the difference between the projections of two elements onto the directional coordinate axis. A distance of 2 indicates a directional influence with a clear direction (one is the source node and the other is the target node), a distance of 1 indicates that only one element has a directional projection and the other is neutral, and a distance of 0 indicates that there is no directional relationship or the directions are parallel. This encoding method transforms discrete directional relationships into numerical representations that can be directly used in vector operations.

[0023] When multiple directed relations are superimposed, the directional basis vectors are expanded into a multi-dimensional direction space, with each dimension encoding an independent type of directed relation. The n different types of directed relations correspond to an n-dimensional direction space, and the projection value of each dimension represents the positive / negative / neutral state of an element under that type of directed relation.

[0024] Compared to directional encoding schemes based on learnable embeddings, the explicit +1 / -1 / 0 encoding of this invention is fundamentally different: learnable embedding schemes maintain a trainable directional embedding vector for each element, and directional information is obtained implicitly through learning; their encoding quality depends on the distribution and scale of the training data. In contrast, the +1 / -1 / 0 encoded values ​​of this invention are directly determined by the directed relation, eliminating the need for a learning process during the training phase, resulting in deterministic and interpretable encoding results. In scenarios with sparse directional relations, learnable embedding schemes struggle to learn effective directional representations with limited samples, while the explicit encoding of this invention is unaffected by sparsity; the +1 / -1 / 0 assignment can be completed with a single scan of the relation graph.

[0025] Example 2: Directional Bias Attention

[0026] Directional basis vectors are incorporated into the attention mechanism through directional bias terms. In the h-th attention head, the attention weight of element i to element j is calculated as follows:

[0027] attn_weight(h,i,j) = softmax(Q_i · K_j^T / √d + α_h · D_ij)

[0028] Where D_ij is the directional distance between elements i and j, and α_h is the learnable scaling parameter of the h-th attention head. α_h is automatically adjusted during training through backpropagation to control the intensity of directional information utilization in attention weight calculation.

[0029] In multi-head attention, different attention heads can be set with different initial α_h values, allowing each head to develop different sensitivities to directional information during training. Some heads may focus on the forward direction (emphasizing the source-target relationship), some on the reverse direction (emphasizing the target-source relationship), and some ignore directional information.

[0030] Example 3: Directional Constraint Loss Function

[0031] The directional constraint loss function adds an extra constraint term to the standard training loss (such as cross-entropy), penalizing predictions that are inconsistent with the known directional relationship:

[0032] L_total = L_base + λ · L_dir

[0033] Where L_dir is the directional constraint term, and λ is the balance coefficient (typically 0.1 to 0.5).

[0034] The directional constraint term L_dir is calculated as follows: For a known directed relation pair (i,j) in the training data (where i has a positive influence on j), the negative logarithm of the predicted probability P_θ(token_j | x) of token_j output by the neural network on the input sequence x is taken. This predicted probability is modulated by the directional bias term α_h·D_ij of the attention enhancement module—in the direction of the known directed relation i→j, the directional bias enhances the model's predicted probability of token_j, while suppressing it in the opposite direction. The sum of the negative logarithmic predicted probabilities of all known relation pairs is L_dir. The essence of this constraint term is to force the model to use the encoded directional information for prediction—if the model correctly uses the directional information (the directional influence of i→j), the predicted probability of token_j output by the model under the known directed relation should be high; otherwise, L_dir increases, and the model parameters are adjusted through backpropagation to make the prediction result consistent with the known directed relation.

[0035] Directional constraints are used only during the training phase and are not involved in the calculation during the test inference phase, thus not affecting the inference speed.

[0036] Example 4: Directed Graph Construction and Storage Circuit

[0037] The directed graph construction and storage circuit automatically constructs a directed graph from the input sequence and maintains the relationship data in hardware through an on-chip adjacency matrix memory. Construction methods include:

[0038] (1) Explicit relation extraction based on trigger words: directional relations in the sequence are detected by a predefined list of directional trigger words (such as "because", "so", "lead to", "therefore", "if" etc.). The elements before and after the trigger words are marked as positive and negative, respectively.

[0039] (2) Implicit inference based on time sequence: In time series data, the preceding time point is marked as the positive influence direction on the subsequent time point, and the subsequent time point is marked as the reverse tracing direction on the preceding time point.

[0040] (3) Importing directional relationships based on external knowledge bases: For data with known directed relational structures (such as directed edges in a knowledge graph), directly import relational pairs.

[0041] The constructed directed graph is stored as an adjacency matrix, with elements taking values ​​{+1, -1, 0}, representing forward, reverse, and no relation, respectively. This matrix corresponds one-to-one with the directional basis vector encoding.

[0042] The storage circuit for the directed relation graph consists of three parts: (i) an adjacency matrix memory, implemented as a two-dimensional register array of shape (L, L), with each register being 2 bits wide (encoding 00 = no relation, 01 = forward, 10 = reverse, 11 = reserved), where L is the maximum sequence length. The memory is row-addressed, with each row corresponding to the directed relation vector between one element in the sequence and all other elements. (ii) a relation update logic, which writes the adjacency matrix row by row when a new sequence is input—for each element pair (i,j), the word matching circuit outputs the direction encoding value, and the enable signal drives the write operation of the corresponding register row. The update operation is completed before the start of the sequence embedding stage and does not occupy the attention computation cycle. (iii) a basis vector encoding interface, which combines the projection values ​​of each row of the adjacency matrix into a directional basis vector tensor: read the column index j corresponding to all non-zero elements in row i, and write the projection value +1 or -1 to the corresponding position of the basis vector tensor. This interface is implemented using pure combinational logic, and the read latency is the propagation time of the comparison chain for non-zero elements in a row, which typically does not exceed a single clock cycle in a typical configuration with L≤512.

[0043] Example 5: Multi-architecture interface circuit for direction encoding module

[0044] The directional encoding module's multi-architecture interface circuit uses a set of configurable data selectors and data format converters to adapt the data path between the directional encoding module and different neural network architectures, supporting addition and concatenation modes. A typical configuration integrates with the Transformer architecture: the directional encoding module and attention enhancement module are embedded in the Transformer's attention layer, and the directional constraint module is invoked during the training loop.

[0045] During the input phase, after the sequence tokens pass through the embedding layer, the directional encoding module generates directional basis vectors based on the directed relation graph, which are then added to or concatenated with the token embeddings. The fused directionally enhanced embeddings are then fed into the standard Transformer layer, but the attention computation is replaced with a version that includes directional bias. During the training phase, after forward propagation of each batch, the directional constraint loss is calculated and weighted and summed with the standard loss, and backpropagation updates all parameters.

[0046] The system integration module includes a multi-architecture interface adapter circuit, supporting data path connections between the directional encoding module and different neural network architectures. The adapter circuit consists of a set of configurable data selectors and data format converters: (i) an embedding layer interface, which receives the feature tensor output by the embedding layer (shape (B, L, d_model)), and fuses it with the basis vectors after the directional basis vectors are encoded using either vector addition or vector concatenation. The addition mode requires d_dir=d_model, while the concatenation mode supports d_dir≠d_model, switched by the mode selection signal of the data selector. (ii) an attention layer interface, which injects the directional bias term α_h·D_ij into the attention fraction tensor as an additional bias term. For architectures using different attention implementation methods (such as the fraction tensor shape of standard multi-head attention being (B, h_num, L, L), the group structure of group attention, etc.), the adapter circuit aligns the directional distance tensor D_ij to the shape of the target attention fraction tensor through broadcasting or repeated padding, ensuring dimensional consistency of the bias injection. (iii) Loss Layer Interface: The calculation result of the directional constraint loss is output in scalar form and merged with the basic loss in the weighted summation unit. The weight coefficient λ of the weighted summation unit is set through a configuration register and remains constant during training. All three interface circuits described above are implemented using pure combinational logic and do not introduce additional clock cycle delays.

[0047] Example 6: Hardware Mapping of Neural Network Accelerators

[0048] This embodiment provides a computational pipeline scheme for deploying directional basis vector encoding and directional bias attention to a neural network accelerator (NPU). The directional encoding module, feature fusion module, attention enhancement module, and directional constraint module serve as the four-level computational units of the accelerator, and are executed in a pipelined manner through data scheduling from on-chip memory.

[0049] The directed graph storage circuit maintains an adjacency matrix representing the directed relationships between elements. The matrix elements have values ​​{+1, -1, 0} and a shape of (L, L), where L is the sequence length. The basis vector encoding circuit reads the directed relation projection values ​​of each element row by row from the adjacency matrix, generating a directional basis vector tensor of shape (L, d_dir), where d_dir is the direction encoding dimension (typically equal to the embedding dimension d_model). The value of each row vector in each dimension is repeatedly filled with the directed relation projection value of the corresponding element under that direction type, and the projection values ​​are normalized to a three-value set of {-1, 0, +1}. The basis vector encoding circuit is implemented using pure combinational logic, requires no clock control, and typically completes the encoding operation of all L elements in a single cycle.

[0050] The feature fusion module receives the feature tensor (shape (B, L, d_model)) output from the embedding layer and the directional basis vector tensor (shape (B, L, d_dir)) output from the directional encoding module. The vector addition circuit performs element-wise addition: enhanced_embedding = embedding + directional_basis. In the typical configuration of d_dir = d_model, the addition operation is a direct element-wise addition of tensors of the same dimension, and the output tensor shape is preserved as (B, L, d_model).

[0051] The attention enhancement module is executed in the matrix multiplication unit of the accelerator. The attention weights are calculated using a version with directional bias:

[0052] The first step is to calculate the dot product of Q and K^T using the matrix multiplication unit. The resulting tensor has the shape (B, h_num, L, L).

[0053] The second step involves the directional distance calculation circuit calculating the directional distance between elements pairwise based on the directional basis vectors, D_ij = |proj(i) - proj(j)|. The directional distance tensor has the shape (B, 1, L, L) and is broadcast to all attention heads.

[0054] The third step is to store the learnable scaling parameter α_h for each attention head h in the scalar parameter register group. The vector processing unit performs element-wise multiplication of α_h and D_ij and then superimposes it onto the corresponding head position of the attention fraction tensor (the scalar parameter register group and the vector processing unit together constitute the attention bias modulation circuit).

[0055] The fourth step is to perform exponential normalization along the last dimension of the attention fraction tensor using the softmax normalization unit.

[0056] The fifth step is to use a weighted summation unit to sum the normalized weights and the value tensor V, and output the directionally enhanced attention result with the shape (B, L, d_model).

[0057] The directional constraint module is activated during the training phase. After the model's forward propagation is complete, the directional constraint loss calculation circuit reads the known set of directed relation pairs R from the predicted output tensor and the directed relation graph storage circuit, and calculates the constraint loss according to the formula L_dir =-Σ_{(i,j)∈R} log P_θ(token_j | x). The constraint loss and the basic loss are combined into the total loss in the weighted summation unit, and the backpropagation unit updates all learnable parameters, including α_h, along the computation graph. During the inference phase, the directional constraint module does not participate in the calculation and does not affect the inference speed.

[0058] Example 7: Typical Configuration Parameters

[0059] When deploying directional basis vector encoding and directional bias attention to a 12-layer Transformer encoder, typical configuration parameters for each module include embedding dimension, number of attention heads, directional encoding dimension, and adjustable range of each item. The specific configuration parameters are as follows.

[0060] Typical configuration parameters for each module are as follows: 12 encoder layers, each containing directional encoding and attention enhancement; embedding dimension d_model is 512, serving as the common dimension for input and output features; directional encoding dimension d_dir is 512, equal to d_model, and can be directly added and fused; number of direction types is 4, supporting both single-dimensional and multi-dimensional direction spaces; total number of attention heads is 8, each with an independent α_h parameter; dimension d_k of each head is 64, equal to d_model divided by the number of attention heads; initial value of α_h is 0.1, automatically adjusted during training; loss balance coefficient λ is 0.1 to 0.5, configurable according to task requirements; batch size B is 16 to 64, configurable according to accelerator memory capacity; maximum sequence length L is 512, configurable from 128 to 2048 during training; directed relation construction methods include trigger word matching, temporal inference, and external knowledge base import, which can be used in combination.

[0061] The parameter configuration described above results in a minimal increase in the number of parameters—the added parameters come only from a learnable scalar α_h for each attention head in each layer (the balancing coefficient λ of the directional constraint loss is a constant configured before training and does not increase the number of parameters), making the increase in parameters relative to a standard 12-layer Transformer negligible. The +1 / -1 / 0 projection values ​​of the directional encoding module are directly determined by the directed relation graph, without introducing additional learnable parameters. This is the essential difference between the directional encoding method of this invention and learnable embedding schemes.

[0062] The above parameters can be adjusted before training according to task requirements: the number of orientation types can be between 1 and 8, the total number of attention heads can be between 4 and 16, and the number of encoder layers can be between 6 and 24. These adjustments do not change the core calculation method of directional basis vector encoding.

[0063] Example 8: Text Causal Chain Extraction

[0064] This embodiment provides a scheme for automatically extracting causal chains from text by applying directional encoding. Directional causal relationships are widely expressed in text, such as causal chains like "A→B→C". The directional information (cause → effect) in these chains is crucial for structured knowledge extraction, but existing attention mechanisms cannot distinguish causal directions, treating causes and effects as equivalent co-occurrence relationships.

[0065] This embodiment constructs a directed relation graph on text summarization data: directed relation pairs between events are extracted based on a causal trigger word library (including "cause", "induce", "promote", "inhibit", "trigger", "block", etc.). Directional basis vectors map cause nodes to +1 and result nodes to -1. The directional constraint loss forces the model to predict the probability of the result given a cause in a known causal chain higher than the probability of the opposite cause. The core advantage of directional encoding is that the explicit encoding of causal direction (+1 / -1 / 0 projection values ​​are directly determined by the directed relation graph, without relying on training) allows the model to naturally distinguish between cause and result nodes in causal chain extraction—standard attention mechanisms treat causal pairs as equally weighted co-occurrence relationships and cannot distinguish directions; the directional constraint loss applies a positive bias in the known causal direction, tending to increase the attention weight of cause nodes towards result nodes and decrease it towards the opposite direction—this mechanism is not dependent on the specific dataset size or distribution and can be effective in any scenario where causal trigger words can cover it.

[0066] Example 9: Modeling the Directional Transmission of Event Sequences

[0067] This embodiment provides a scheme for applying directional encoding to model directional transmission relationships in event sequences. There are explicit directional transmission relationships between event sequences, such as a transmission chain of "source → target 1 → target 2 → target 3". The direction of such a chain is determined by the underlying logic, but traditional time-series models (such as LSTM and Transformer) treat event sequences as equally weighted contexts, failing to distinguish between source and target nodes in the transmission chain.

[0068] The data processed in this embodiment is a structured event stream, where each event contains three basic pieces of information: event type, timestamp, and direction of influence. The directed relationship graph is constructed as follows: based on the logical connections between event types, prior propagation rules are defined (e.g., "Event A → Event B", "Event B → Event C"). Nodes representing the preceding event are encoded as +1, and nodes representing the succeeding event are encoded as -1. Directional basis vectors are superimposed during the event embedding stage, enabling the model to naturally perceive the direction of propagation during attention calculation.

[0069] The advantage of directional encoding in modeling directional transmission lies in maintaining the transmission direction: standard attention mechanisms treat event sequences as equally weighted contexts, failing to distinguish between source and target nodes in the transmission chain; directional encoding, by adding directional information to each event node during the embedding stage through +1 / -1 projection values, allows the attention mechanism to naturally perceive the transmission direction—the attention weights from the source node to the target node are amplified by the directional bias term, while the attention weights from the target node to the source node are suppressed. In multi-layered transmission chains (such as "source → target 1 → target 2 → target 3"), the directional bias accumulates layer by layer, enabling the model to maintain directional consistency even in long transmission chains. Directional constraint loss further focuses on the consistency of transmission direction: if the model predicts event B follows event A, but the known directional relationship between A and B is B→A (opposite direction), then the directional constraint loss imposes a penalty—this mechanism makes the model tend to choose predictions consistent with the known transmission direction when the predicted direction of the event type is ambiguous, regardless of the specific dataset.

[0070] The special role of directional constraint loss in this scenario lies in the consistency of the transmission chain: unlike the standard cross-entropy loss which focuses on the accuracy of event type prediction, directional constraint loss pays additional attention to the consistency of the transmission direction, making the model tend to choose the prediction result consistent with the known transmission direction when the event type prediction direction is ambiguous.

[0071] The directional encoding method of the present invention only requires adding a directional basis vector superposition operation in the input embedding stage, without introducing additional model parameters. In the inference stage, only a small number of tensor operations are introduced for directional distance calculation and bias superposition, making it suitable for directional sequence modeling on resource-constrained devices.

[0072] Example 10: Optional Deployment on Non-Binary Radix Arithmetic Units

[0073] Optionally, in this embodiment, the basis vector encoding circuit of the directional encoding module is deployed on a non-binary radix arithmetic unit. This hardware deployment is not a necessary implementation of the directional encoding mechanism of this invention—the directional encoding module of this invention can also be implemented in software or firmware on a general-purpose matrix operation unit or a general-purpose processor. The non-binary radix arithmetic unit uses radix-5 as its base, and its structured opcode decoder employs bit selection logic—the highest bit distinguishes between arithmetic and bit operation classes, and the second highest bit is the numerator class, directly connected to the opcode bit lines through combinational logic gates. The +1 / -1 / 0 basis vector encoding of the directional encoding module and the bit selection logic share the same set of signal bit lines: the bit selection signal lines of the basis vector encoding are multiplexed with the bit lines of the opcode decoder, simultaneously completing opcode decoding and directional basis vector encoding in the combinational logic gate array. During deployment, the non-binary radix arithmetic unit typically completes the encoding output of the directional basis vector within a single clock cycle, and the basis vector encoding delay does not add an extra clock cycle. The +1, -1, and 0 values ​​in the basis vector encoding are directly mapped to the three encoding intervals of the opcode bit lines, eliminating the need for an additional encoding lookup table. The basis vector encoding circuit itself achieves bit selection signal sharing within the same combinational logic gate array through bit line multiplexing with the opcode decoder. The basis vector encoding is implemented using pure combinational logic, without adding any extra clock cycle delay.

[0074] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A neural network training system based on directional coding, characterized in that, include: The directional encoding module includes a directed relation graph storage circuit and a basis vector encoding circuit. The directed relation graph storage circuit stores the adjacency matrix {+1,-1,0} of directed relations between elements. The input of the basis vector encoding circuit is connected to the output of the directed relation graph storage circuit, and the output outputs the explicit encoded values ​​of the forward basis vector +1 and the reverse basis vector -1. The explicit encoded values ​​are directly determined by the directed relations rather than obtained through learning. The forward and reverse basis vectors constitute a direction coordinate axis, and the projection values ​​of elements on the direction coordinate axis are taken from the three-value set {+1,-1,0}. The attention enhancement module includes a directional distance calculation circuit and an attention bias modulation circuit. The input of the directional distance calculation circuit is connected to the output of the directional coding module to calculate the directional distance D_ij between elements. The attention bias modulation circuit multiplies the directional distance D_ij with the learnable scaling parameter α_h and then adds it to the attention score, so that the attention distribution is modulated by directional information. The directional constraint module includes a directional constraint loss calculation circuit. Its input is connected to the model prediction output and the set R of known directed relation pairs. The directional constraint loss is calculated according to the formula L_dir = -Σ_{(i,j)∈R} log P_θ(token_j | x), where P_θ(token_j | x) is the predicted probability of token_j output by the neural network on the input sequence x. The predicted probability is enhanced in the direction of the known directed relation i→j by the directional bias term of the attention enhancement module. The output is connected to the weighted summation unit. The feature fusion module includes a vector addition circuit or a vector concatenation circuit. Its first input terminal is connected to the feature vector output terminal of the embedding layer, and its second input terminal is connected to the directional basis vector output terminal of the directional encoding module. The output terminal outputs the directional enhanced embedding vector.

2. The system according to claim 1, characterized in that, It also includes a non-binary radix arithmetic unit, which uses positive integers other than powers of 2 as its base, and includes a structured opcode decoder. The structured opcode decoder uses bit selection logic to be directly connected to the opcode bit lines through combinational logic gates, without the need for a lookup table or read-only memory. The basis vector encoding circuit and the bit selection logic of the non-binary radix arithmetic unit share the bit lines, and simultaneously complete opcode decoding and direction basis vector encoding in the same combinational logic gate array.

3. The system according to claim 1, characterized in that, The directional encoding module and attention enhancement module are embedded in the attention layer of the Transformer architecture, and the directional constraint module is called in the training loop; the basis vector encoding circuit is implemented with pure combinational logic and does not require clock control.

4. A sequence relation encoding method based on directional basis vectors, applied to the system described in claim 1, characterized in that, Executed on a computing device including a neural network accelerator (NPU), the NPU including a matrix multiplication unit, a vector processing unit, and a softmax normalization unit, the method includes: Obtain the directed relationship information between elements in the sequence, wherein the directed relationship information represents the directional influence of the first element on the second element; Directed relationships between elements are automatically detected from the input sequence, and the detected directed relationships are organized into a directed graph structure. The directed relationships include relationships indicated by directional trigger words and implicit inference relationships based on temporal order. The directed relation information is encoded as directional basis vectors, which include forward basis vectors and reverse basis vectors. The forward basis vectors encode the unidirectional influence direction of the first element on the second element, and the reverse basis vectors encode the reverse tracing direction of the second element on the first element. The directional basis vectors are fused with the feature representations of the elements in the sequence to generate directional-enhanced embedding vectors. The directionally enhanced embedding vector is input into a neural network to perform sequence modeling tasks.

5. The method according to claim 4, characterized in that, The encoding rule for the directional basis vectors is as follows: The positive basis vector is assigned the first sign value +1, which represents the unidirectional propagation direction from the source element to the target element; The reverse basis vector is assigned the second sign value -1, which indicates the reverse tracing direction from the target element to the source element; The forward and reverse basis vectors form a one-dimensional direction coordinate axis, and the directional distance is defined as the absolute value of the difference between the projection values ​​of the two elements onto the direction coordinate axis.

6. The method according to claim 4, characterized in that, The process of fusing directional basis vectors with element feature representations includes: A directional bias term is introduced into the attention mechanism of a neural network, which is calculated based on the directional distance between elements; The output weights of the attention mechanism depend on both the similarity of element features and the strength of the directed relation: attn_weight(h,i,j) = softmax(Q_i · K_j^T / √d + α_h · D_ij) Where D_ij is the directional distance between element i and element j, and α_h is the learnable scaling parameter of the h-th attention head.

7. The method according to claim 4, characterized in that, It also includes a directional constraint loss function: A directional constraint term L_dir is added to the training objective of the neural network. This directional constraint term penalizes predictions that are inconsistent with known directional relationships. L_total = L_base + λ · L_dir L_dir = -Σ_{(i,j)∈R} log P_θ(token_j | x) Where R is the set of known directed pairs, P_θ(token_j | x) is the predicted probability of the neural network outputting token_j under the modulation of the directional bias term α_h·D_ij, and λ is the balance coefficient.

8. The method according to claim 5, characterized in that, The explicit +1 / -1 / 0 encoded values ​​are directly determined by the directed relation and are not learned from training data; the directional basis vectors are generated in the input stage through a single scan of the directed relation graph, and the encoding results are deterministic and interpretable.

9. The method according to claim 4, characterized in that, The directional encoding is executed on a hardware platform containing a non-binary radix arithmetic unit. The non-binary radix arithmetic unit uses a positive integer other than a power of 2 as its base. Its structured opcode decoder uses bit selection logic to be directly connected to the opcode bit line through combinational logic gates. The basis vector encoding and the bit selection logic of the structured opcode decoder share the bit line, and opcode decoding and directional basis vector encoding are completed simultaneously in the same combinational logic gate array.

Images