A hybrid attention sequence processing method and system based on dynamic routing and background state updating

Abstract

This invention discloses a hybrid attention sequence processing method and system based on dynamic routing and background state updates. The method includes: receiving input features and performing normalization processing; based on the processed input features, performing the following actions: updating the recursive hidden state of the linear attention expert branch; generating a compressed latent vector and writing it into the global cache of the sparse attention expert branch; generating a routing gating signal based on the processed input features, dynamically activating either the linear attention expert branch or the sparse attention expert branch; if the former is activated, calculating the linear output based on the updated recursive hidden state; if the latter is activated, starting an indexer to filter K historical information from the global cache, restoring them as keys to calculate the sparse output; and performing a residual concatenation between the linear output or the sparse output and the input features to obtain the final output. This invention can dynamically and seamlessly switch between the two branches and effectively maintain the continuity of the sequence state.

Images