Data processing method and device based on hybrid attention

Abstract

The application discloses a data processing method and device based on a hybrid attention. The method comprises the following steps: when a current word element in a word element sequence corresponding to to-be-processed data is processed, a target historical word element set corresponding to the current word element is determined through a full attention mechanism in a hybrid attention network; state attenuation information of a linear attention mechanism in the hybrid attention network is determined according to the correlation between the current word element and a target historical word element in the target historical word element set; and the historical recursive state of the linear attention mechanism is updated according to the state attenuation information, so that a current recursive state of the linear attention mechanism corresponding to the current word element is obtained. According to the application, the target historical word element screened by the full attention mechanism is used to optimize the state attenuation information of the linear attention mechanism, thereby improving the updating accuracy of the historical recursive state to the long-sequence key historical information.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a data processing method and apparatus based on hybrid attention. Background Technology

[0002] As attention-based large models are increasingly used in scenarios such as long sequence data processing, image understanding, speech recognition, text generation, and multimodal tasks, the length of sequences that these models need to process is constantly increasing. For example, in scenarios such as long-context text understanding, retrieval-enhanced generation, long video analysis, and long audio recognition, models often need to extract information related to the current object from a relatively long historical sequence.

[0003] Standard full attention mechanisms can model historical information relatively accurately based on the correlation between current and historical words. However, as the sequence length increases, the amount of attention-related information that standard full attention mechanisms need to compute and store also increases, resulting in high computational and storage costs when processing long sequences.

[0004] To reduce the processing cost of long sequences, existing technologies have proposed linear attention mechanisms and hybrid attention networks combining linear and full attention mechanisms. Linear attention mechanisms typically use historical recursion to compress and model historical information, reducing computational and storage overhead in long sequence processing. Hybrid attention networks, on the other hand, can, to some extent, balance the processing efficiency of linear attention mechanisms with the modeling capabilities of full attention mechanisms.

[0005] However, the historical recursive states in linear attention mechanisms are typically represented in a compressed form, which has relatively limited capacity. When updating historical recursive states, it's necessary to control the retention and decay of historical information. If the state decay information cannot accurately reflect the correlation between the current word and key historical words, it may lead to excessive decay of key historical information related to the current word, or the continued retention of historical information with low relevance to the current word in the historical recursive state. These issues will affect the modeling performance of linear attention mechanisms on key historical information in long sequences, and consequently, the model's performance in scenarios involving long-context text, long videos, long audio, or multimodal data processing.

[0006] Therefore, how to improve the accuracy of retaining and controlling the decay of key historical information during the historical recursive state update process while maintaining the low computational and storage overhead of the linear attention mechanism is a technical problem that needs to be solved. Summary of the Invention

[0007] In order to solve at least one of the technical problems in the background art described above, this application proposes a data processing method and apparatus based on hybrid attention.

[0008] One aspect of this application provides a data processing method based on hybrid attention, the method comprising: When processing the current word in the word sequence corresponding to the data to be processed, the target historical word set corresponding to the current word is determined by the full attention mechanism in the hybrid attention network; Based on the correlation between the current word and the target historical word in the target historical word set, the state decay information of the linear attention mechanism in the hybrid attention network is determined; Based on the state decay information, the historical recursive state of the linear attention mechanism is updated to obtain the current recursive state of the linear attention mechanism corresponding to the current word.

[0009] Optionally, the data processing method based on hybrid attention further includes: Based on the key vector and value vector corresponding to the target historical word in the target historical word set, the target key vector and target value vector corresponding to the current word in the linear attention mechanism are determined. The step of updating the historical recursive state of the linear attention mechanism based on the state decay information includes: The historical recursive state is updated based on the state decay information, the target key vector, and the target value vector.

[0010] Optionally, determining the target historical word set corresponding to the current word unit through the full attention mechanism in a hybrid attention network includes: For each of the multiple rating dimensions, the dimensional correlation between the current word element and each historical word element in that rating dimension is determined based on the query vector of the current word element in that rating dimension and the key vectors corresponding to each historical word element. Based on the state update information of the current word element corresponding to the linear attention mechanism, determine the weight coefficient corresponding to this scoring dimension; Based on the weight coefficients corresponding to the multiple scoring dimensions, the dimensional relevance of the same historical word element under the multiple scoring dimensions is weighted to obtain the selection score corresponding to the historical word element. The target set of historical lexical units is determined based on the selection scores corresponding to each historical lexical unit.

[0011] Optionally, the hybrid attention network includes a first full attention layer, n linear attention layers, and a second full attention layer connected in sequence, where n is an integer greater than or equal to 1; At least one of the n linear attention layers is followed by a merging layer; The attention output of the first full attention layer is transmitted to the adjacent linear attention layer and the merging layer; The merging layer merges the attention output of the first full attention layer and the attention output of the corresponding linear attention layer to obtain a merged output, and then transmits the merged output to the next network layer.

[0012] Optionally, the hybrid attention network includes m linear attention layers connected in sequence and a third full attention layer connected in parallel with the m linear attention layers, where m is an integer greater than or equal to 1; The third full attention layer is used to determine the target historical word set corresponding to the current word, and to transmit the target historical word set to at least one of the m linear attention layers; The linear attention layer that receives the target historical word set updates the historical recursive state based on the target historical word set.

[0013] Optionally, updating the historical recursive state of the linear attention mechanism based on the state decay information to obtain the current recursive state of the linear attention mechanism corresponding to the current word includes: Based on the query vector and value vector corresponding to the current word, determine the state update information of the linear attention mechanism; Based on the state decay information and the state update information, the historical recursive state is updated to obtain the current recursive state.

[0014] Optionally, determining the target key vector and target value vector corresponding to the current word in the linear attention mechanism based on the key vector and value vector corresponding to the target historical word in the target historical word set includes: Based on the correlation between each target historical word in the target historical word set and the current word, the selected historical word is determined from the target historical word set; The key vector and value vector corresponding to the selected historical word are determined as the target key vector and the target value vector.

[0015] Optionally, the data processing method based on hybrid attention further includes: Based on the query vector corresponding to the target historical word in the target historical word set, the target query vector corresponding to the current word in the linear attention mechanism is determined; Based on the current recursive state and the target query vector, the attention output corresponding to the current word in the linear attention mechanism is generated.

[0016] Optionally, determining the weight coefficients corresponding to the scoring dimension based on the state update information of the current lexical unit corresponding to the linear attention mechanism includes: Based on the state update strength information of the current word in the linear attention mechanism, the value vector strength information of the current word in the scoring dimension, and the key vector strength information corresponding to the scoring dimension, the weight coefficient corresponding to the scoring dimension is determined.

[0017] Optionally, each of the n linear attention layers is followed by the merging layer; The attention output of the first full attention layer is transmitted to each of the merging layers, and each merging layer merges the attention output of the first full attention layer and the attention output of the corresponding linear attention layer.

[0018] Optionally, the third full attention layer transmits the corresponding target historical word set to each of the m linear attention layers.

[0019] Optionally, for the target linear attention layer among the m linear attention layers, the third full attention layer determines the target historical word set corresponding to the target linear attention layer based on the state update information corresponding to the target linear attention layer, and transmits the corresponding target historical word set to the target linear attention layer.

[0020] Another aspect of this application provides a data processing apparatus based on hybrid attention, the apparatus comprising: The target historical word set determination unit is used to determine the target historical word set corresponding to the current word in the word sequence corresponding to the data to be processed by using the full attention mechanism in the hybrid attention network. The state decay information determination unit is used to determine the state decay information of the linear attention mechanism in the hybrid attention network based on the correlation between the current word and the target historical word in the target historical word set. The recursive state update unit is used to update the historical recursive state of the linear attention mechanism according to the state decay information, so as to obtain the current recursive state of the linear attention mechanism corresponding to the current word.

[0021] To achieve the above objectives, according to another aspect of this application, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described data processing method based on hybrid attention.

[0022] To achieve the above objectives, according to another aspect of this application, a computer-readable storage medium is also provided, on which a computer program / instructions are stored, which, when executed by a processor, implement the steps of the above-described data processing method based on hybrid attention.

[0023] To achieve the above objectives, according to another aspect of this application, a computer program product is also provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method based on hybrid attention.

[0024] The beneficial effects of this application are as follows: In this embodiment, when processing the current word element, a set of target historical word elements corresponding to the current word element is first determined through a full attention mechanism. Then, the state decay information of the linear attention mechanism is determined based on the correlation between the current word element and the target historical word elements in the target historical word element set. The historical recursive state of the linear attention mechanism is then updated based on this state decay information. Thus, when updating the historical recursive state, the linear attention mechanism can refer to the correlation between the target historical word elements selected by the full attention mechanism and the current word element, making the state decay information more reflective of the degree of association between the current word element and key historical word elements, thereby improving the accuracy of the current recursive state in representing key historical information. In long-context text question answering or text generation scenarios, this method helps to retain long-distance historical word element information related to the current word element. In long video, long audio, or multimodal sequence processing scenarios, this method helps to improve the representation ability of the current recursive state for key historical information across frames, segments, or modalities, reducing the risk of excessive decay of key historical information during the recursive state update process. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 This is a flowchart of a data processing method based on hybrid attention, as described in an embodiment of this application. Figure 2This is a flowchart illustrating how the full attention mechanism in this application determines the target historical word set corresponding to the current word. Figure 3 This is a flowchart illustrating how the historical recursive state is updated according to an embodiment of this application; Figure 4 This is a flowchart illustrating how the target key vector and target value vector of the current word in the linear attention mechanism are determined according to an embodiment of this application. Figure 5 This is a flowchart illustrating the attention output of the linear attention mechanism corresponding to the current word in an embodiment of this application. Figure 6 This is a first schematic diagram of the first architecture of the hybrid attention network in this application; Figure 7 This is a second schematic diagram of the first architecture of the hybrid attention network in this application; Figure 8 This is a first schematic diagram of the second architecture of the hybrid attention network in this application; Figure 9 This is a second schematic diagram of the second architecture of the hybrid attention network in this application; Figure 10 This is a structural block diagram of a data processing device based on hybrid attention, according to an embodiment of this application. Figure 11 This is a schematic diagram of a computer device according to an embodiment of this application. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0027] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0028] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification, claims and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product or device.

[0029] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The present application will now be described in detail with reference to the accompanying drawings and embodiments.

[0030] Before providing a detailed description of the embodiments of this application, a general overview of the relevant attention mechanisms and hybrid attention architecture will be given.

[0031] Many large models based on the Transformer architecture employ the standard attention mechanism, also known as full attention (FA). The full attention mechanism can adequately model the correlation between the current word and historical words, thus exhibiting strong information selection capabilities in long context understanding, text generation, retrieval-augmented generation (RAG), agent tasks, image understanding, long audio recognition, long video analysis, and multimodal tasks. However, when dealing with long sequences, the full attention mechanism typically requires attention calculations between the current word and a large number of historical words, along with storing the corresponding historical key-value information, leading to high computational and storage costs.

[0032] Various optimization methods have been proposed for the full attention mechanism. For example, to reduce the storage overhead of key-value information, different forms such as multi-head attention (MHA), grouped-query attention (GQA), and multi-head latent attention (MLA) can be adopted. To reduce the computational cost of attention scores, sparsity methods such as sliding window attention (SWA) and top-k selection can be used. These optimization methods can reduce the computational or storage cost of the full attention mechanism to some extent, but in long sequence scenarios, a trade-off between modeling capability and processing cost still needs to be struck.

[0033] To further reduce the cost of processing long sequences, linear attention (LA) mechanisms have been proposed in related technologies. LA mechanisms typically compress and model historical information using historical recursive states, resulting in lower computational and storage overhead compared to standard full attention mechanisms, making them more suitable for processing long-contextual data. However, the capacity of historical recursive states is limited, and the information that can be stored is constrained by the size of the compressed states. When the retention and decay of historical information cannot be accurately controlled during the update process of historical recursive states, it may affect their ability to express key historical information.

[0034] To balance the modeling capabilities of full attention mechanisms with the processing efficiency of linear attention mechanisms, hybrid attention architectures have been proposed in related technologies. One approach involves sequentially setting up several linear attention layers and full attention layers according to a certain quantitative relationship, such as setting up a full attention layer after multiple linear attention layers. Another approach involves setting up linear attention branches and full attention branches in parallel and fusing the outputs of the two types of branches. The sequential hybrid architecture has lower overall computational complexity, but the linear attention layers and full attention layers usually rely mainly on the inter-layer output transmission, and the information collaboration between the two is still relatively limited. The parallel hybrid architecture can utilize the outputs of both linear attention mechanisms and full attention mechanisms simultaneously, but it usually introduces additional computational overhead, and the simple fusion method cannot fully leverage the complementary roles of the two types of attention mechanisms in historical information compression and key historical information selection.

[0035] Based on the above, this application provides a data processing method based on hybrid attention. This method does not simply connect or parallel linear attention and full attention mechanisms; instead, it further utilizes the full attention mechanism's ability to select target historical lexical units to assist the linear attention mechanism in updating the historical recursive state. This improves the retention and attenuation control effect of the linear attention mechanism on key historical information in long sequence scenarios. The processing procedure of this application embodiment is described below with reference to specific embodiments.

[0036] Figure 1 This is a flowchart of a data processing method based on hybrid attention, as described in an embodiment of this application. Figure 1 As shown, in one embodiment of this application, the data processing method based on hybrid attention includes steps S101 to S103.

[0037] Step S101: When processing the current word in the word sequence corresponding to the data to be processed, the target historical word set corresponding to the current word is determined by the full attention mechanism in the hybrid attention network.

[0038] In this application, the data to be processed may include at least one of text data, image data, audio data, video data, and multimodal data. A token can be understood as a basic sequence unit in the model processing. For text data, a token can be a coding unit corresponding to a character, word, sub-word, or text segment; for image data, a token can be a coding unit corresponding to an image patch, image region, or image feature; for audio data, a token can be a coding unit corresponding to an audio frame, audio segment, or acoustic feature; for video data, a token can be a coding unit corresponding to a video frame, video segment, spatiotemporal block, or video feature; for multimodal data, a token can be a sequence unit obtained after encoding different modal features.

[0039] The current lexicon refers to the lexicon that currently requires attention processing. Historical lexicons refer to those preceding the current lexicon in the lexicon sequence, which can be used as a reference for the current lexicon. The target historical lexicon set refers to the set of historical lexicons determined from the historical lexicons and used to assist in the processing of the current lexicon. The target historical lexicon set may include one or more target historical lexicons. The target historical lexicon set can be represented by at least one of the following: position, index, identifier, selection score, attention weight, key vector, and value vector. In this embodiment, determining the target historical lexicon set does not require transmitting or saving the original data of the target historical lexicons themselves.

[0040] In this embodiment, Full Attention (FA) can be used to determine the relevance between the current word and historical words. Based on the standard full attention relevance calculation method, the full attention mechanism determines the attention score or relevance score between the current word and historical words, and determines the target historical word set through sparse selection, Top-k selection, or other filtering methods. That is, the full attention mechanism in this embodiment is not limited to an attention structure that directly outputs the target historical word set; in some embodiments, the relevance between the current word and historical words can be calculated first based on the full attention mechanism, and then the target historical word set can be filtered from the historical words based on this relevance.

[0041] In optional embodiments of this application, the full attention mechanism can be a standard multi-head attention mechanism (MHA), or an attention mechanism obtained by performing key-value compression, grouping, de-ranking, or sparsification on the full attention mechanism. For example, the full attention mechanism can be a grouped query attention (GQA), a multi-head latent attention (MLA), or a sparse attention mechanism with Top-k selection. When using a standard multi-head attention mechanism, the attention score obtained from the standard multi-head attention mechanism can be further filtered to determine the target historical word set; when using a sparse attention mechanism with Top-k selection, the target historical word set can be determined based on the Top-k selection result.

[0042] In one embodiment of this application, this step can sort historical words based on the relevance between the current word and each historical word, and select at least a portion of the top-ranked historical words as the target historical word set. In other embodiments, the target historical word set can also be determined from historical words based on a relevance threshold, selection score, attention weight, or preset filtering rules. This embodiment does not limit the specific method of determining the target historical word set, as long as the target historical word set can represent at least a portion of historical words related to the current word.

[0043] Step S102: Determine the state decay information of the linear attention mechanism in the hybrid attention network based on the correlation between the current word and the target historical word in the target historical word set.

[0044] In this application, Linear Attention (LA) refers to an attention mechanism that models historical information through recursive states, state spaces, or compressed states. Linear Attention can compress the information of historical words using historical recursive states, thereby reducing computational and storage overhead in long sequence processing. Historical recursive states can be in vector, matrix, or tensor form, representing the historical information accumulated before processing the current word.

[0045] In this embodiment, the relevance between the current word and the target historical word can be determined based on the degree of matching between the query vector corresponding to the current word and the key vector corresponding to the target historical word, or it can be determined based on attention score, similarity score, selection score, distance metric, or gating score. The relevance can be in scalar, vector, matrix, or tensor form, and this application does not limit it in this regard.

[0046] In this application, state decay information is used to control the degree to which the linear attention mechanism retains or decays existing historical information when updating historical recursive states. State decay information may include a state decay factor, a gating decay factor, a decay gating vector, decay weights, or other information used to adjust the degree of decay of historical recursive states. State decay information can be set for the entire historical recursive state, or it can be set separately for different dimensions, channels, or state components within the historical recursive state.

[0047] In this embodiment, a correlation feature can be generated based on the correlation between the current word and each target historical word in the target historical word set, and state decay information can be determined based on this correlation feature. For example, the correlations corresponding to each target historical word can be weighted, normalized, nonlinearly mapped, or gated to obtain feature information used to determine the state decay information. The state decay information can be used to adjust the retention ratio or decay intensity of existing historical information in the historical recursive state, so that the update process of the historical recursive state can refer to the degree of correlation between the current word and the target historical word.

[0048] In step S102, the state decay information no longer depends solely on the recursive state of the linear attention mechanism or the information of the current word itself. Instead, it can be determined by combining the correlation between the target historical word set determined by the full attention mechanism and the current word. In this way, the state decay information can more directly reflect the degree to which the current word depends on key historical information in the target historical words.

[0049] Step S103: Update the historical recursive state of the linear attention mechanism according to the state decay information to obtain the current recursive state of the linear attention mechanism corresponding to the current word.

[0050] In this application, the historical recursive state is the state information formed by the linear attention mechanism before processing the current word, and the current recursive state is the updated state obtained by the linear attention mechanism after processing the current word. The current recursive state can be used to generate the attention output corresponding to the current word, or it can continue to be used in the processing of subsequent words.

[0051] In this embodiment, the existing historical information in the historical recursive state can be attenuated according to the state decay information, and the attenuated historical recursive state can be updated by combining the information corresponding to the current word to obtain the current recursive state. In other embodiments, the historical recursive state can also be updated by combining the key vector, value vector or other representation information corresponding to the target historical word set.

[0052] In this embodiment, after obtaining the current recursive state, the linear attention mechanism can generate a linear attention output corresponding to the current word based on the current recursive state. Specifically, the linear attention mechanism can determine the output representation corresponding to the current word based on the current recursive state and the query vector corresponding to the current word. The output representation can be used as the input to subsequent network layers in the hybrid attention network, or it can be fused with the output of the full attention mechanism before being transmitted to subsequent network layers.

[0053] In existing linear attention mechanisms, the degree of state decay is typically determined by decay parameters preset by the model or learned, or it can be obtained through projection functions, gating functions, or nonlinear mappings based on the input features corresponding to the current word, used to control the retention or decay of existing historical information in the historical recursive state. Compared to existing methods that primarily determine the degree of state decay based on the linear attention mechanism's own parameters or the current word's own features, this application uses a full attention mechanism to determine the target historical word set corresponding to the current word, and determines the state decay information based on the correlation between the current word and the target historical word set. This allows the linear attention mechanism to refer to more targeted historical relevance information when updating the historical recursive state. Therefore, it can improve the accuracy of updating key historical information in long sequences by the current recursive state. This approach can be applied to data processing scenarios such as long contextual text, long audio, long video, and multimodal sequences, effectively improving the accuracy of the current recursive state in representing key historical information in these scenarios.

[0054] like Figure 3 As shown, in one embodiment of this application, step S103 above updates the historical recursive state of the linear attention mechanism according to the state decay information to obtain the current recursive state of the linear attention mechanism corresponding to the current word, including steps S301 and S302.

[0055] Step S301: Determine the state update information of the linear attention mechanism based on the query vector and value vector corresponding to the current word.

[0056] In this embodiment, the query vector corresponding to the current word element can be used to represent the current word element's need to read or match historical information, and the value vector corresponding to the current word element can be used to represent the content information carried by the current word element itself. State update information is used to adjust the intensity of the information corresponding to the current word element's participation in historical recursive state updates. State update information can be in scalar, vector, matrix, or tensor form, and this application does not limit its application to this form.

[0057] In relevant linear attention mechanisms, state update information can serve as a gating parameter, learning rate parameter, or update intensity parameter during the state update process, controlling the degree of influence of the information corresponding to the current word on the historical recursive state. For example, in some linear attention mechanisms, state update information can participate in the directional correction of the historical recursive state and in the writing process of the information corresponding to the current word. Unlike existing technologies that determine the state update intensity solely through preset parameters or a single input mapping, this application embodiment can combine the query vector and value vector corresponding to the current word to determine the state update information, enabling the state update information to simultaneously reflect the historical information matching requirements of the current word and the content intensity carried by the current word itself.

[0058] In this embodiment, feature extraction, concatenation, weighting, norm calculation, nonlinear mapping, or gating mapping can be performed on the query vector and value vector corresponding to the current word element to obtain state update information. For example, the update strength of the current word element to the historical recursive state can be determined based on the strength information of the query vector and the value vector. Alternatively, the query vector and value vector can be input into a gating function to obtain a gating value used to control the degree of state update. The larger the state update information, the stronger the writing or updating effect of the information corresponding to the current word element on the historical recursive state. The smaller the state update information, the weaker the writing or updating effect of the information corresponding to the current word element on the historical recursive state.

[0059] This step allows us to determine state update information based on the query and content features of the current lexical unit. This enables the linear attention mechanism to consider not only the decay of existing historical information but also the contribution of the current lexical unit information to the update of the historical recursive state when updating the historical recursive state.

[0060] Step S302: Based on the state decay information and the state update information, update the historical recursive state to obtain the current recursive state.

[0061] In this embodiment, state decay information is used to control the degree of retention or decay of existing historical information in the historical recursive state, while state update information is used to control the intensity of the participation of the information corresponding to the current word in the historical recursive state update. In other words, state decay information can be used to adjust the overall retention or decay of existing information in the historical recursive state, while state update information can be used to adjust the intensity of correction, erasure, or writing to the historical recursive state based on the information of the current word. Both participate in the historical recursive state update process.

[0062] In this embodiment, the historical recursive state can be decayed based on state decay information, and the information corresponding to the current word can be written into or merged into the historical recursive state based on state update information to obtain the current recursive state. In other embodiments, the state decay information and state update information can be applied together to the historical recursive state update function, and the current recursive state can be output through the historical recursive state update function. This application does not limit the specific state update formula.

[0063] Through this step, the linear attention mechanism can control the degree of retention or decay of historical information, as well as the strength of the participation of current lexical information in state updates. Compared to updating historical recursive states using only fixed recursion rules or a single gating parameter, this embodiment can determine state decay information based on the correlation between the current lexical and the target historical lexical, and complete the state update by combining it with the state update information corresponding to the current lexical. This makes the current recursive state more accurately reflect the historical information that needs to be retained and the current information that needs to be written during the current lexical processing.

[0064] In a preferred embodiment of this application, the full attention mechanism in the hybrid attention network can employ a multi-head latent attention (MLA) mechanism with Top-k sparse selection. This MLA mechanism can select a set of target historical words that are highly correlated with the current word based on the correlation between the current word and historical words. The linear attention mechanism in the hybrid attention network can employ a gated delta network (GDN). This GDN can compress and model historical word information using historical recursive states and update the historical recursive states based on state decay information.

[0065] It should be noted that this application uses the MLA mechanism with Top-k sparse selection as an example of a full attention mechanism and GDN as an example of a linear attention mechanism, mainly to illustrate the collaborative processing method between the full attention mechanism and the linear attention mechanism in this application. This application is not limited to the specific attention mechanisms described above. In other embodiments, the full attention mechanism may also employ other full attention or sparse full attention mechanisms capable of determining the target historical word set corresponding to the current word; the linear attention mechanism may also employ other linear attention mechanisms capable of compressing and modeling historical information through historical recursive states.

[0066] The mechanism of Gated Incremental Network (GDN) is briefly introduced below: Gated Incremental Networks (GDNs) can recursively represent the key-value association information in historical lexical units through a state matrix, and control the update process of the state matrix through a gating decay factor and a state update factor.

[0067] For example, the state update process of GDN can be represented as:

[0068] Among them, S t S represents the state matrix corresponding to the t-th lexical, which can also be understood as a memory matrix used to store historical key-value association information; t-1 I represents the state matrix after the (t-1)th word has been processed; I represents the identity matrix; q t k represents the query vector corresponding to the t-th term; t This represents the key vector (Key) corresponding to the t-th word; v t Represents the value vector (Value) corresponding to the t-th word; O t,GDN This represents the attention output of the GDN corresponding to the t-th lexical.

[0069] α t This represents the gating attenuation factor, used to control the degree to which historical state information is retained or attenuated. When α t When α is small, the decay of historical state information is greater, which is beneficial for cleaning up information in historical states that no longer needs to be retained; when α is small... t As the value approaches 1, the global decay effect of historical state information weakens, and GDN focuses more on updating the state matrix based on the key vector and value vector corresponding to the current word during the state update process.

[0070] β t β represents the state update factor, used to adjust the intensity of the update of the state matrix by the key-value information corresponding to the t-th word. Specifically, β t It can participate in the directional correction of existing key-value associations in the state matrix, and participate in the adjustment of the value vector v corresponding to the current word. t and key vector k t The resulting key-value associations are written into the state matrix.

[0071] It should be noted that the above formula is only an exemplary expression of the GDN state update process. In other embodiments, the gating decay factor α t and state update factor β t It can also be in scalar, vector, or matrix form; query vector q t Key vector k t Sum vector v t It can also be transformed, normalized, or projected according to the specific structure of different linear attention mechanisms. This application does not limit the specific parameterization form of GDN.

[0072] In this embodiment of the application, when the linear attention mechanism uses GDN, the historical recursive state in the above embodiment can correspond to the state matrix formed by GDN before processing the current word, for example, it can correspond to S in the above formula. t-1 The current recursive state in the above embodiments can correspond to the state matrix obtained by GDN after processing the current word, for example, it can correspond to S in the above formula. t The state decay information in the above embodiments can correspond to the gating decay factor α in the GDN. t This is used to control the degree to which historical information is retained or decayed in the historical recursive state. The state update information in the above embodiments can correspond to the state update factor β in the GDN. t This is used to adjust the intensity of the update of the state matrix based on the key value information corresponding to the current word. Therefore, the process of updating the historical recursive state based on state decay information and state update information can be represented in the GDN example as based on α... t and β t The process of updating the state matrix.

[0073] The following is a brief introduction to the mechanism of Multi-Head LatentAttention (MLA) with Top-k sparse selection: MLA can jointly compress key and value vectors using low-rank latent representations to reduce the amount of key-value information that needs to be cached during inference, thereby reducing storage overhead when processing long sequences.

[0074] For example, let h be the embedding vector of the t-th token. t h t ∈R d The lower projection matrix is ​​W. DKV W DKV ∈R dc×d , where d c The dimension is smaller than the dimension of the key-value vector after multi-head expansion. The latent key-value representation corresponding to the t-th lexical can be obtained by the following formula:

[0075] in, Let be the key-value compressed representation corresponding to the t-th word. Further, this can be achieved through an up-projection matrix. Reconstruction yields compressed key and value vectors for attention computation:

[0076] Among them, W UK W is the up projection matrix corresponding to the key vector. UVThe upper projection matrix corresponding to the value vector. This refers to the key vector portion reconstructed from the key-value compressed representation. This is the value vector portion reconstructed from the key-value compressed representation.

[0077] In some embodiments, to incorporate positional information, a position-dependent key vector portion can also be generated. For example, the key vector portion generated from the embedding vector can be position-encoded using Rotary Position Embedding (RoPE) to obtain:

[0078] Based on this, the reconstructed key vector part can be... Location-related key vector part By concatenating the vectors, we obtain the key vector corresponding to the t-th word:

[0079] The value vector corresponding to the t-th word can be the part of the value vector reconstructed above, that is:

[0080] For the current word, MLA can calculate the attention score between the current word and historical words based on the query vector corresponding to the current word and the key vector corresponding to historical words. For example, for the t-th word and the j-th historical word, under the i-th attention head, the attention score can be determined as follows:

[0081] Where, q t,i Let k represent the query vector of the t-th word under the i-th attention head. j,i Let d represent the key vector of the j-th historical word under the i-th attention head. h This represents the content dimension of each attention head. α represents the dimension of the location-related key vector part. t,j,i This represents the attention weight of the t-th word element to the j-th historical word element under the ith attention head.

[0082] Furthermore, the output corresponding to the i-th attention head can be obtained based on the attention weights and value vector:

[0083] in, represents the value vector part of the j-th historical token under the i-th attention head. The outputs of multiple attention heads can be concatenated and passed through an output projection matrix to obtain the attention output corresponding to the t-th token of MLA:

[0084] where W O is the output projection matrix, n h is the number of attention heads.

[0085] In the MLA mechanism with Top-k sparse selection, it is not necessary to perform full attention calculation on all historical tokens. Instead, selection scores are calculated for the current token and the historical tokens first, and part of the historical tokens are selected from the historical tokens according to the selection scores. Exemplarily, for the position t where the current token is located and the historical position s, where s<t, the selection score can be calculated in the following manner:

[0086] where I t,s represents the selection score between the current token and the s-th historical token, H I represents the number of scoring dimensions, represents the query representation of the current token under the j-th scoring dimension, represents the key representation corresponding to the s-th historical token, represents the weight coefficient corresponding to the j-th scoring dimension. Based on the selection scores corresponding to each historical position, K historical positions with higher selection scores can be selected, and the historical tokens corresponding to the K historical positions are determined as the target historical token set corresponding to the current token.

[0087] In the embodiments of the present application, when the full attention mechanism adopts the multi-head latent attention mechanism MLA with Top-k sparse selection, the current token in the above embodiments may correspond to the token corresponding to the current query position in MLA, and the historical token in the above embodiments may correspond to the token corresponding to the historical position located before the current query position in MLA. The MLA mechanism with Top-k sparse selection may determine the selection score corresponding to each historical token based on the correlation between the current token and each historical token, and select historical tokens satisfying the Top-k condition from the historical tokens according to the selection scores. Accordingly, the target historical token set in the above embodiments may correspond to the set formed by the historical tokens satisfying the Top-k condition. That is to say, the process of determining the target historical token set corresponding to the current token through the full attention mechanism in the aforementioned step S101 may be embodied in this embodiment as: determining the selection score corresponding to each historical token through the MLA mechanism with Top-k sparse selection, and determining the historical tokens whose selection scores satisfy the Top-k condition as the target historical token set corresponding to the current token.

[0088] It should be noted that the historical word set obtained by Top-k selection is only one example of the target historical word set. In other embodiments, the target historical word set can also be determined based on selection score thresholds, attention weights, relevance ranking, or other sparse selection rules. This application does not limit the target historical word set to be determined by a fixed number of Top-k methods.

[0089] In one specific embodiment of this application, the full attention mechanism in the hybrid attention network adopts a multi-head latent attention (MLA) mechanism with top-k sparse selection, and the linear attention mechanism adopts a gated delta network (GDN). In this embodiment, MLA is used to determine the target historical word set corresponding to the current word from historical words, and GDN is used to update the historical recursive state based on the relevance information corresponding to the target historical word set.

[0090] Specifically, for the current word t, MLA can determine the Top-K set from historical words based on the correlation between the current word t and each historical word. t Top-K set t This can include selected historical positions, where the historical lexical units corresponding to those positions can serve as the target historical lexical units corresponding to the current lexical unit t. Therefore, in this embodiment, the Top-K set is Top-K. t The corresponding historical lexical units can form the target historical lexical unit set in the aforementioned step S101.

[0091] For the current word t, the state update process of GDN can be represented as:

[0092] The attention output of GDN for the current word t can be represented as:

[0093] Among them, S t-1 To process the historical recursive state matrix before the current word t; S t Let I be the updated current recursive state matrix; I is the identity matrix; k t and v t These are the key vector and value vector corresponding to the current word; α t β is the gating attenuation factor used to control the retention or attenuation of historical information; t q is the state update factor, used to adjust the intensity of the current lexical information being written into the state matrix; tu is the query vector corresponding to the current term. t,GDN This is the attention output for GDN.

[0094] In this embodiment, the aforementioned state decay information can be specifically embodied in the gating decay factor α in the GDN. t Gating attenuation factor α t Based on the current word t and the Top-K set Top-K t The correlation between historical lexical units is determined. For example, it can be determined using the following formula:

[0095] in, Let be the Top-K historical word set of the current word t, determined by MLA; σ is the sigmoid function; K represents the number of historical positions in the Top-K set; s represents the historical positions in the Top-K set, and k s This represents the key vector corresponding to the historical position s, and sim() represents the similarity function.

[0096] In some embodiments, sim() can use a cosine similarity function, which can be specifically expressed as:

[0097] Therefore, the gating attenuation factor α t This reflects the overall correlation between the current word 't' and the target historical words corresponding to the Top-K set. When the overall correlation between the target historical words corresponding to the Top-K set and the current word 't' is high, the gating decay factor α... t This allows GDN to improve the retention of relevant historical state information during state updates; when the overall correlation between the target historical lexicon corresponding to the Top-K set and the current lexicon t is low, the gating decay factor α... t This can improve the attenuation of historical state information in GDN.

[0098] In this embodiment, the aforementioned state update information may include the state update factor β in the GDN. t State update factor β t Based on the query vector q corresponding to the current word t t Sum vector v t Determine. For example, it can be determined using the following formula:

[0099] State update factor β t This is used to adjust the intensity of the update of the key value information corresponding to the current word t to the historical recursive state. Specifically, β tIt can participate in the targeted correction of existing key-value associations in the historical recursive state, and participate in the transformation of the value vector q corresponding to the current word t. t and key vector v t The resulting key-value associations are written to the historical recursive state.

[0100] Based on the above formula, the process of determining the target historical word set corresponding to the current word in step S101 can be specifically manifested as follows: determining the Top-K set through MLA with Top-K sparse selection, and using the historical words corresponding to the Top-K set as the target historical word set. The process of determining the state decay information in step S102 can be specifically manifested as follows: determining the gate decay factor α in the GDN based on the similarity between the current word t and the target historical words in the target historical word set. t The process of updating the historical recursive state based on the state decay information in step S103 can be specifically described as follows: based on the gated decay factor α t and state update factor β t The historical recursive state S of GDN t-1 Update the current recursive state S. t .

[0101] Using the above method, the Top-K selection results of MLA can be used to determine the gating attenuation factor α of GDN. t This allows GDN to consider the correlation between the current lexicon and the target historical lexicon when updating the historical recursive state. Thus, GDN's retention and decay of historical state information no longer relies solely on its own fixed decay rules or the information of the current lexicon itself, but can be dynamically adjusted in conjunction with the target historical lexicon selected by MLA, thereby improving the accuracy of the historical recursive state in updating key historical information.

[0102] In the above embodiments, the hybrid attention network can first determine the target historical word set corresponding to the current word through a full attention mechanism, and then determine the state decay information of the linear attention mechanism based on the correlation between the current word and the target historical word set, thereby improving the accuracy of historical information retention and decay control in the historical recursive state. Furthermore, in another embodiment of this application, the target historical word set can also be used to assist in determining the key vector and value vector used by the linear attention mechanism during state update. That is, when updating the historical recursive state, the linear attention mechanism can not only determine the state decay information based on the target historical word set, but also select key vectors and value vectors more suitable for the current word processing needs from the historical information corresponding to the target historical word set, and update the historical recursive state based on the selected key vectors and value vectors. This can further improve the utilization effect of the linear attention mechanism on the relevant local historical information of the current word.

[0103] In one embodiment of this application, the data processing method based on hybrid attention further includes: Based on the key vector and value vector corresponding to the target historical word in the target historical word set, the target key vector and target value vector corresponding to the current word in the linear attention mechanism are determined. In one embodiment of this application, updating the historical recursive state of the linear attention mechanism based on the state decay information in step S103 above includes: The historical recursive state is updated based on the state decay information, the target key vector, and the target value vector.

[0104] In this application, the target historical word set is determined by a full attention mechanism. Since the full attention mechanism can identify target historical words with a high degree of correlation with the current word based on the relevance between the current word and historical words, the target historical words in the target historical word set typically provide more valuable historical information for the current word. The key vector corresponding to the target historical word can be used to represent the index features of the target historical word in the attention matching process, and the value vector corresponding to the target historical word can be used to represent the content information carried by the target historical word.

[0105] In this embodiment, a selected historical word can be determined from the target historical word set based on the correlation between each target historical word and the current word, and the key vector and value vector corresponding to the selected historical word can be determined as the target key vector and the target value vector. For example, the historical word with the highest correlation to the current word in the target historical word set can be selected as the selected historical word, and the key vector and value vector corresponding to the selected historical word can be used as the target key vector and the target value vector.

[0106] In other embodiments of this application, the target key vector and the target value vector can also be determined based on the key vectors and value vectors corresponding to multiple target historical lexical units in the target historical lexical unit set. For example, the key vectors corresponding to each target historical lexical unit can be weighted according to the correlation between each target historical lexical unit in the target historical lexical unit set and the current lexical unit to obtain the target key vector; and the value vectors corresponding to each target historical lexical unit can be weighted to obtain the target value vector. Alternatively, the key vectors and value vectors corresponding to multiple target historical lexical units in the target historical lexical unit set can be concatenated, pooled, weighted summed, attention-based, or linearly projected to obtain the target key vector and target value vector. The embodiments of this application do not limit the specific method of determining the target key vector and target value vector, as long as the target key vector and target value vector can be determined based on the key vector and value vector corresponding to at least one target historical lexical unit in the target historical lexical unit set.

[0107] In one embodiment of this application, updating the historical recursive state of the linear attention mechanism based on the state decay information in step S103 above includes: updating the historical recursive state based on the state decay information, the target key vector, and the target value vector.

[0108] In this embodiment, state decay information is used to control the degree of retention or decay of existing historical information in the historical recursive state, and the target key vector and target value vector are used to provide historical key-value information related to the current word. When updating the historical recursive state, the existing information in the historical recursive state can be decayed first according to the state decay information, and then the decayed historical recursive state can be updated based on the target key vector and target value vector; alternatively, the state decay information, target key vector, and target value vector can be input together into the state update function to obtain the updated current recursive state.

[0109] In this embodiment, when the linear attention mechanism employs Global Density Nodes (GDN), the target key vector can be used to replace or modify the key vector in the GDN state update process, and the target value vector can be used to replace or modify the value vector in the GDN state update process. That is, when updating the historical recursive state, GDN does not rely entirely on the key and value vectors corresponding to the current lexical unit itself, but can further introduce the target key and value vectors determined by the target historical lexical unit set. Therefore, the GDN state update process can make greater use of the key historical information filtered by the full attention mechanism.

[0110] For example, when the full attention mechanism uses the MLA mechanism with Top-K sparse selection and the linear attention mechanism uses GDN, the MLA mechanism can first determine the Top-K historical word set corresponding to the current word. Then, based on the key vectors and value vectors corresponding to the historical words in this Top-K historical word set, the target key vector and target value vector used in the GDN state update process can be determined. In this way, the more valuable historical key-value information selected by the MLA mechanism can participate in the recursive state update process of GDN, thereby enhancing GDN's ability to utilize local historical information related to the current word.

[0111] In this way, the target historical lexical set is not only used to determine state decay information, but also further used to determine the target key vector and target value vector used by the linear attention mechanism when updating the historical recursive state. Compared to the linear attention mechanism's method of updating the state using only the key vector and value vector of the current lexical itself, this embodiment can introduce historical key-value information more relevant to the current lexical into the historical recursive state update process, thereby improving the linear attention mechanism's utilization of historical information related to the current lexical.

[0112] like Figure 4 As shown, in one embodiment of this application, the above steps of determining the target key vector and target value vector of the linear attention mechanism corresponding to the current word based on the key vector and value vector corresponding to the target historical word in the target historical word set include steps S401 and S402.

[0113] Step S401: Based on the correlation between each target historical word in the target historical word set and the current word, determine the selected historical word from the target historical word set.

[0114] Step S402: Determine the key vector and value vector corresponding to the selected historical word as the target key vector and the target value vector.

[0115] In this embodiment of the application, the correlation between each target historical word in the target historical word set and the current word can be determined first. The correlation can be the selection score, attention score, similarity score or attention weight obtained by the aforementioned full attention mechanism when determining the target historical word set, or it can be the correlation recalculated based on the current word and the target historical word.

[0116] In this embodiment, the target historical word element with the highest relevance to the current word element in the target historical word element set can be determined as the selected historical word element. In other embodiments, the target historical word element with the highest ranking in the target historical word element set can also be selected as the selected historical word element based on the relevance ranking result. Alternatively, a target historical word element can be determined as the selected historical word element from the target historical word elements with a relevance exceeding a preset threshold.

[0117] like Figure 5 As shown, in one embodiment of this application, the data processing method based on hybrid attention further includes steps S501 and S502.

[0118] Step S501: Determine the target query vector corresponding to the current word in the linear attention mechanism based on the query vector corresponding to the target historical word in the target historical word set.

[0119] In this embodiment, the target query vector refers to the query vector used to generate the attention output corresponding to the current word in combination with the current recursive state. The target query vector can be determined based on the query vector corresponding to at least one target historical word in the target historical word set.

[0120] In this embodiment, a selected historical word can be determined from the target historical word set based on the correlation between each target historical word and the current word, and a target query vector can be determined based on the query vector corresponding to the selected historical word. For example, the query vector corresponding to the selected historical word can be used as the target query vector, or the query vector corresponding to the selected historical word can be linearly projected, normalized, or gated to obtain the target query vector.

[0121] In some embodiments of this application, the query vectors corresponding to multiple target historical lexical units can be weighted according to the correlation between each target historical lexical unit and the current lexical unit to obtain the target query vector. For example, the query vectors corresponding to multiple target historical lexical units can be weighted and summed based on attention score, selection score, similarity score, or normalized weight to obtain the target query vector.

[0122] In other embodiments of this application, the query vector corresponding to the current word element itself can be fused with the query vector obtained based on the target historical word element set to obtain the target query vector. For example, the query vector corresponding to the current word element itself and the query vector corresponding to the target historical word element set can be concatenated, weighted summed, gated fusion, or linearly projected to obtain the target query vector. Thus, the target query vector can both retain the query requirements of the current word element itself and incorporate historical query features related to the current word element from the target historical word element set.

[0123] Step S502: Based on the current recursive state and the target query vector, generate the attention output of the linear attention mechanism corresponding to the current word.

[0124] In this embodiment, the current recursive state can be the state information obtained by the linear attention mechanism after processing the current word, which includes historical information updated by state decay information. The target query vector can be used to read the state information related to the current word from the current recursive state to generate the attention output corresponding to the current word.

[0125] In this embodiment, when the linear attention mechanism uses GDN, the current recursive state can correspond to the state matrix in the GDN. In this case, the state matrix can be read based on the target query vector to obtain the attention output of the current word corresponding to the linear attention mechanism. For example, matrix-vector operations can be performed between the current recursive state and the target query vector to obtain the output representation corresponding to the current word. This embodiment does not limit the specific output calculation formula, as long as the attention output can be generated based on the current recursive state and the target query vector.

[0126] Currently, linear attention mechanisms typically read the current recursive state based on the query vector corresponding to the current word element to generate the attention output for that word element. In contrast, this application's embodiment can determine the target query vector based on the query vector corresponding to the target historical word element in the target historical word element set. This allows the linear attention mechanism to further reference historical query features related to the current word element when reading the current recursive state. This helps improve the accuracy of the linear attention mechanism in reading historical information related to the current word element, enabling the generated attention output to better reflect the relationship between the current word element and key historical word elements.

[0127] In the above embodiments, this application mainly utilizes the target historical word set determined by the full attention mechanism to optimize the linear attention mechanism, enabling the linear attention mechanism to make fuller use of the key historical information filtered by the full attention mechanism. Furthermore, in another embodiment of this application, the state update-related information in the linear attention mechanism can also be used in reverse to optimize the determination process of the target historical word set by the full attention mechanism. Specifically, when determining the target historical word set, not only the instantaneous correlation between the current word and historical words can be considered, but also the contribution of historical information demonstrated by the linear attention mechanism during the state update process can be combined, thus making the selected target historical words more consistent with the update requirements of the current recursive state. Therefore, the bias caused by filtering solely based on the direct correlation between the current word and historical words can be reduced, improving the accuracy of the selection of the target historical word set.

[0128] like Figure 2 As shown, in one embodiment of this application, the step S101 above, which determines the target historical word set corresponding to the current word through the full attention mechanism in the hybrid attention network, includes steps S201 to S204.

[0129] Step S201: For each of the multiple rating dimensions, determine the dimensional correlation between the current word and each historical word in the rating dimension based on the query vector of the current word in that rating dimension and the key vectors corresponding to each historical word.

[0130] Scoring dimensions can be understood as different computational dimensions used to evaluate the relevance between the current lexical unit and historical lexical units. Different scoring dimensions can correspond to different attention heads, different projection spaces, different scoring channels, or different relevance calculation methods. The query vector of the current lexical unit under this scoring dimension can be the query representation obtained by projecting, mapping, or splitting the query vector of the current lexical unit. The key vector corresponding to historical lexical units can be the key vector of the historical lexical unit, or the key representation obtained by projecting or mapping the key vector of the historical lexical unit.

[0131] Dimensional relevance is used to represent the degree of matching between a current term and a historical term within a given rating dimension. In some embodiments, dimensional relevance under a rating dimension can be determined based on the dot product, normalized dot product, similarity calculation result, or nonlinear mapping result between the query vector of the current term and the key vector corresponding to the historical term within that rating dimension. For example, the dot product between the query vector of the current term and the key vector corresponding to the historical term can be calculated first, and then activation processing can be applied to this dot product result to obtain the dimensional relevance.

[0132] Step S202: Determine the weight coefficient corresponding to the scoring dimension based on the state update information of the current word corresponding to the linear attention mechanism.

[0133] In this application, state update-related information refers to information related to the linear attention mechanism's updating of historical recursive states. State update-related information can be used to characterize the linear attention mechanism's update requirements for different information dimensions, different state components, or different key-value information when processing the current lexical unit. For example, state update-related information may include at least one of the following: state update strength information, value vector strength information corresponding to the current lexical unit, key vector strength information, gating information, and update weight information.

[0134] In this embodiment, the weighting coefficient is used to represent the importance of the corresponding scoring dimension in the target historical word selection process. That is, among multiple scoring dimensions, the degree of influence of different scoring dimensions on the target historical word selection result can be different. By determining the weighting coefficients corresponding to the scoring dimensions based on the state update information of the linear attention mechanism, the full attention mechanism can consider not only the matching relationship between the current word and the historical word when selecting the target historical word, but also the state update requirements of the linear attention mechanism.

[0135] Step S203: Based on the weight coefficients corresponding to the multiple scoring dimensions, the dimensional relevance of the same historical word element under the multiple scoring dimensions is weighted to obtain the selection score corresponding to the historical word element.

[0136] In this embodiment, the full attention mechanism, for any historical word element, first determines the dimensional relevance of that historical word element under multiple scoring dimensions. Then, using the weight coefficients corresponding to each scoring dimension, it performs weighted summation, weighted averaging, or other weighted fusion processing on the dimensional relevance of the historical word element under multiple scoring dimensions to obtain the selection score corresponding to that historical word element. This selection score is used to characterize the priority of the historical word element being selected into the target historical word element set.

[0137] In one embodiment of this application, the selection score corresponding to the historical term s can be expressed as:

[0138] Among them, I t,s H represents the selection score between the current lexicon t and the historical lexicon s; I This indicates the number of rating dimensions; j represents the rating dimension. This represents the weight coefficient of the current word element t under the scoring dimension j; This represents the query representation of the current word element t under the scoring dimension j; represents the key representation corresponding to the historical term s; ReLU represents the activation function. The above formula is only one example; in other embodiments, other activation functions, normalization functions, or weighted fusion methods can also be used to determine the selection score.

[0139] Step S204: Determine the target historical word set based on the selection score corresponding to each historical word.

[0140] In this embodiment, the full attention mechanism can sort historical lexical units according to their selection scores, and select at least a portion of the historical lexical units with higher selection scores as the target historical lexical unit set corresponding to the current lexical unit. For example, the top K historical lexical units in the selection score ranking can be determined as the target historical lexical unit set. Alternatively, historical lexical units whose selection scores meet a selection score threshold can be selected as the target historical lexical unit set. This embodiment does not limit the specific selection rules for the target historical lexical unit set.

[0141] Through steps S201 to S204, the process of determining the target historical word set is not simply a matter of equally weighted or fixed-weighted fusion of relevance across multiple scoring dimensions, but rather a further introduction of state update-related information from the linear attention mechanism. This allows the influence of historical words on the state update process of the linear attention mechanism to be considered when selecting target historical words, making the historical words selected for the target historical word set more aligned with the update requirements of the current recursive state. Consequently, the accuracy of the target historical word set selection can be improved, and a more reliable historical information foundation can be provided for the subsequent determination of state decay information and the updating of historical recursive states.

[0142] In one embodiment of this application, step S202 above, which determines the weight coefficient corresponding to the scoring dimension based on the state update information of the current word unit corresponding to the linear attention mechanism, includes: Based on the state update strength information of the current word in the linear attention mechanism, the value vector strength information of the current word in the scoring dimension, and the key vector strength information corresponding to the scoring dimension, the weight coefficient corresponding to the scoring dimension is determined.

[0143] In this embodiment, the state update strength information is used to characterize the strength of the linear attention mechanism's update of the historical recursive state when processing the current lexical unit. In some embodiments of this application, when the linear attention mechanism employs GDN, the state update strength information may include the state update factor in the aforementioned state update information, for example, it may include the state update factor β from GDN. t The greater the state update strength information, the stronger the effect of the information corresponding to the current word element on updating the historical recursive state.

[0144] In this embodiment, value vector strength information is used to characterize the information strength carried by the value vector of the current word in the corresponding scoring dimension. Optionally, the value vector strength information can be determined based on the norm of the value vector of the current word in that scoring dimension. Key vector strength information is used to characterize the strength of the key vector corresponding to that scoring dimension. For example, the key vector strength information can be determined based on the norm of the key vector corresponding to that scoring dimension.

[0145] In this embodiment, the state update strength information, value vector strength information, and key vector strength information can be multiplied, weighted, nonlinearly mapped, or normalized to obtain the weight coefficients corresponding to the scoring dimension. As a specific example, the weight coefficients corresponding to the scoring dimension can be determined as follows:

[0146] in, βt represents the weight coefficient of the current word t under the scoring dimension j; βt represents the state update strength information of the linear attention mechanism for the current word t. This represents the value vector of the current word element t under the scoring dimension j; This represents the L2 norm of the value vector representation; This represents the key vector representation corresponding to the rating dimension j; This represents the L2 norm of the key vector representation.

[0147] In this way, the weight coefficients corresponding to the scoring dimension are not only related to the correlation between the current word and historical words, but also reflect the update intensity of the linear attention mechanism in the current state update process, as well as the information intensity of the value vector and key vector under this scoring dimension. Thus, when subsequently determining the selection score corresponding to historical words, the influence of the scoring dimension, which is more relevant to the linear attention state update, can be increased, making the selection result of the target historical word set more consistent with the state update requirements of the linear attention mechanism.

[0148] As can be seen from the above embodiments, this application does not simply connect or parallel the full attention mechanism and the linear attention mechanism, but rather enables them to cooperate synergistically in information selection, state updating, and output generation. Specifically, the full attention mechanism can determine the target set of historical words related to the current word from historical words, while the linear attention mechanism can determine state decay information based on the correlation between the current word and the target set of historical words, and update the historical recursive state based on this state decay information. Thus, while maintaining the advantages of recursive state compression modeling, the linear attention mechanism can utilize the selection results of key historical words by the full attention mechanism to improve the accuracy of historical information retention and decay control in the historical recursive state.

[0149] Furthermore, in some embodiments, the target historical lexical set can also be used to determine the target key vector, target value vector, and / or target query vector used by the linear attention mechanism during state updates. Thus, when updating the historical recursive state or generating the attention output corresponding to the current lexical, the linear attention mechanism no longer relies solely on the key, value, or query information corresponding to the current lexical itself, but can further incorporate historical lexical information related to the current lexical. Compared to methods that only rely on the current lexical's own information for state updates and readings, this application can improve the utilization effect of the linear attention mechanism on the historical information related to the current lexical, making the current recursive state and attention output more accurately reflect the association between the current lexical and key historical lexicals.

[0150] Furthermore, in some embodiments, the state update information of the current word in the linear attention mechanism can also be used in reverse to participate in the process of determining the target historical word set by the full attention mechanism. That is, when selecting target historical words, not only the correlation between the current word and historical words is considered, but the update requirements reflected in the state update process of the linear attention mechanism can also be taken into account to weight the correlation under different scoring dimensions. Therefore, the determination result of the target historical word set can better adapt to the historical recursive state update process of the linear attention mechanism, reducing the potential bias that may occur when selecting historical words based solely on instantaneous correlation.

[0151] Therefore, this application achieves bidirectional synergy between the full attention mechanism and the linear attention mechanism, enabling the key historical word selection capability of the full attention mechanism to serve the state update and output generation of the linear attention mechanism. Simultaneously, it allows the state update requirements of the linear attention mechanism to influence the selection process of the target historical word. Compared to existing technologies that simply concatenate, parallel, or directly fuse the two types of attention outputs, this application improves the accuracy of selecting, retaining, updating, and retrieving key historical information in long sequence processing while maintaining the low computational and storage overhead of the linear attention mechanism. This solution is applicable to data processing scenarios such as long contextual text, long audio, long video, and multimodal sequences.

[0152] The above embodiments mainly illustrate the collaborative processing method between the full attention mechanism and the linear attention mechanism from the perspective of information interaction and state update. In actual network structures, the above collaborative processing method can be implemented by combining different hybrid attention architectures. That is, the full attention mechanism and the linear attention mechanism can be set according to different inter-layer connection relationships or branch connection relationships, and the determination of the target historical word set, the determination of state decay information, and the update of historical recursive states can be completed in the corresponding architecture. The implementation method of the embodiments of this application will be further explained below with reference to several specific hybrid attention network architectures.

[0153] Figure 6 This is a first schematic diagram of the first architecture of the hybrid attention network in this application. Figure 6 As shown, in one embodiment of this application, the hybrid attention network may employ a serial hybrid architecture. Figure 6 In this context, LA represents the linear attention layer, FA represents the full attention layer, and "merge" represents the merging layer. The linear attention layer can compress and model historical information using historical recursion states, the full attention layer can model the correlation between the current word and historical words, and the merging layer is used to fuse the attention outputs of the full attention layer and the linear attention layer.

[0154] In one embodiment of this application, the hybrid attention network includes a first full attention layer, n linear attention layers, and a second full attention layer connected sequentially, where n is an integer greater than or equal to 1. "Connected sequentially" here can be understood as follows: the output of the first full attention layer is transmitted along the network processing direction to the subsequent linear attention layers; the n linear attention layers process the data sequentially; and the processing results are then transmitted to the second full attention layer. The first full attention layer can be... Figure 6 The FA located before multiple merging layers can be the second full attention layer. Figure 6 The FA is located after multiple linear attention layers.

[0155] In one embodiment of this application, a merging layer is provided after at least one of the n linear attention layers. The merging layer can be located on the output side of the linear attention layer to receive its attention output. Simultaneously, the attention output of the first full attention layer can also be transmitted to the merging layer. That is, in addition to being transmitted along the main serial path to the adjacent linear attention layer, the attention output of the first full attention layer can also be transmitted via a bypass path to the merging layer located after the subsequent linear attention layer.

[0156] In one embodiment of this application, the merging layer merges the attention output of the first full attention layer and the attention output of the corresponding linear attention layer to obtain a merged output, and transmits the merged output to subsequent network layers. The corresponding linear attention layer can be a linear attention layer located before the merging layer and whose attention output is input to the merging layer. The merging method can include addition, weighted summation, post-projection, gated fusion, or other methods capable of fusing two attention outputs; this application does not limit this method. Subsequent network layers can be subsequent linear attention layers, full attention layers, normalization layers, feedforward network layers, or other network processing layers.

[0157] exist Figure 6 In the illustrated embodiment, the attention output of the first full attention layer can be transmitted to multiple merging layers. Each merging layer merges the attention output of the first full attention layer and the attention output of the corresponding linear attention layer. Thus, the attention output of the first full attention layer can be reused in subsequent linear attention layers, rather than being passed layer by layer along the main serial path. In this way, the relevance modeling information provided by the first full attention layer can be used to enhance the ability of subsequent linear attention layers to express local key historical information.

[0158] In one embodiment of this application, each of the n linear attention layers is followed by a merging layer. In this case, the attention output of the first full attention layer can be transmitted to each merging layer, and each merging layer merges the attention output of the first full attention layer with the attention output of the corresponding linear attention layer. Thus, the output of each linear attention layer can be fused with the attention output of the first full attention layer, enabling subsequent linear attention layers to utilize the relevance modeling information provided by the first full attention layer.

[0159] Compared to a simple serial hybrid architecture, this embodiment does not simply arrange the full attention layer and the linear attention layer sequentially. In a simple serial hybrid architecture, the output of the full attention layer is typically passed as input to the next layer, making it difficult for subsequent linear attention layers to directly utilize the attention modeling information in the full attention layer's output. This embodiment, by setting a merging layer after at least one linear attention layer and transmitting the attention output of the first full attention layer to the merging layer, allows the attention output of the first full attention layer to be fused with the attention output of subsequent linear attention layers. This enhances the ability of subsequent linear attention layers to utilize the output information of the full attention layer.

[0160] Compared to a simple parallel fusion architecture, this embodiment retains the serial processing path of the linear attention layer and performs fusion at subsequent linear attention layers by reusing the attention output of the first full attention layer. This improves the ability of the hybrid attention network to represent key historical information without significantly increasing the number of full attention computations.

[0161] therefore, Figure 6 The first architecture shown introduces a cross-layer reuse mechanism for the output of the first full attention layer on top of the serial hybrid architecture, enabling the fusion of the correlation modeling information provided by the full attention layer and the recursive state information provided by the linear attention layer in subsequent network layers. This architecture helps to improve the problem of insufficient utilization of full attention information in a purely serial architecture and enhances the transmission and expression of key historical information in long sequence data processing.

[0162] Figure 7 This is a second schematic diagram of the first architecture of the hybrid attention network in this application. (See diagram below.) Figure 7 As shown, in Figure 6 Based on the illustrated embodiment, the hybrid attention network may further include multiple serial-parallel fusion units. Each serial-parallel fusion unit may include an attention amplifier (FA), at least one attention lower bound (LA) disposed after the FA, and a merging layer disposed after the corresponding LA. In each serial-parallel fusion unit, the attention output of the FA may be transmitted to at least one subsequent merging layer and merged with the attention output of the corresponding LA.

[0163] exist Figure 7 In the illustrated embodiment, multiple LAs connected sequentially can be set before the first FA. For example, Figure 7 Three serially connected least squares (LAs) are set before the first analytical framework (FA). By setting multiple LAs before the first FA, historical information can be recursively modeled using the LAs first, and then correlation modeling can be performed by subsequent FAs. It should be noted that the number of LAs before the first FA is not limited. Figure 7The three shown can be set to other numbers in other embodiments, depending on the model size, computing resources, or task requirements.

[0164] Furthermore, Figure 7 The diagram illustrates two serial-parallel fusion units. This means that the hybrid attention network can be repeatedly configured at different network depths. Figure 6 The cross-layer merging structure shown allows the attention outputs of different attention layers to be reused at one or more subsequent attention layers (LAs). In this way, the relevance modeling information provided by the full attention layer can be fused with the recursive modeling information of the linear attention layer at multiple network stages.

[0165] pass Figure 7 The structure shown allows the hybrid attention network to retain the sequential processing path dominated by the attentional logic (LA) to control computational overhead during long-sequence processing. Simultaneously, it introduces cross-layer reuse of the attentional logic (FA) outputs at different network locations, enabling the FA's attentional outputs to participate in fusion multiple times at subsequent LA stages. Compared to structures that fuse only at a single location, this embodiment enhances the synergy between FA and LA outputs across multiple network stages, thereby improving the hybrid attention network's ability to transmit and express key historical information.

[0166] Figure 8 This is a first schematic diagram of the second architecture of the hybrid attention network in this application. (See diagram below.) Figure 8 As shown, in one embodiment of this application, the hybrid attention network can adopt a sparse full attention parallel architecture. Figure 8 In this architecture, LA represents a linear attention layer and FA represents a full attention layer. Multiple LAs are connected sequentially to form the main linear attention processing path; the FA is connected in parallel with the multiple LAs as an auxiliary selection branch, used to determine the target historical lexical set corresponding to the current lexical for one or more LAs.

[0167] In this embodiment, the LA can compress and model historical information through historical recursion states to obtain global summary information with lower overhead. The FA can select target historical words that are more relevant to the current word based on the correlation between the current word and historical words, thereby providing more granular historical information selection results. By setting the FA in parallel with multiple serial LAs, the FA's ability to select key historical words can be introduced on the basis of the main path of serial LA processing.

[0168] In one embodiment of this application, the hybrid attention network includes m linear attention layers connected in sequence and a third full attention layer connected in parallel with the m linear attention layers, where m is an integer greater than or equal to 1. "Connected in sequence" here can be understood as: the output of the previous LA is transmitted to the next LA, and multiple LAs are processed sequentially along the network processing direction. "Connected in parallel" here can be understood as: the third full attention layer receives input information that is the same as or related to the m LAs, and determines the target historical word set corresponding to the current word based on this input information; the output of the third full attention layer is not used as the ordinary next-layer output in the main serial path, but is transmitted as auxiliary information to at least one LA.

[0169] In one embodiment of this application, a third full attention layer is used to determine the target historical word set corresponding to the current word, and to transmit the target historical word set to at least one of the m linear attention layers. Here, "transmitting the target historical word set" can be understood as transmitting information used to characterize the target historical word set, such as at least one of the target historical word's position, index, identifier, selection score, attention weight, key vector, and value vector, and is not limited to transmitting the original data of the target historical word itself.

[0170] In one embodiment of this application, the LA that receives the target historical word set can update the historical recursive state based on the target historical word set. For example, the LA can determine state decay information based on the correlation between the current word and the target historical word in the target historical word set, and update the historical recursive state based on the state decay information; alternatively, it can determine the target key vector and target value vector used in the state update process based on the key vector and value vector corresponding to the target historical word in the target historical word set. Thus, the third full attention layer can provide more targeted historical word selection results for one or more LAs, enabling the LA to utilize the key historical information filtered by the FA during the recursive modeling process.

[0171] In one embodiment of this application, the third full attention layer can transmit the corresponding target historical word set to each of the m linear attention layers. That is, each LA can receive the target historical word set determined by the third full attention layer and update its own historical recursion state based on this target historical word set. The target historical word sets received by each LA can be the same or different.

[0172] In some embodiments of this application, the third full attention layer can determine the same set of target historical lexical units for multiple LAs. For example, when multiple LAs have similar requirements for selecting historical information, the third full attention layer can determine a common set of target historical lexical units based on the correlation between the current lexical unit and historical lexical units, and transmit this common set of target historical lexical units to the multiple LAs respectively. At this time, each LA can perform historical recursive state updates based on the same set of target historical lexical units, but different LAs can still form different current recursive states based on their own state decay information, state update information, or state update function.

[0173] In other embodiments of this application, the third full attention layer can also determine different target historical word sets for different LAs. Specifically, for the target linear attention layer among the m linear attention layers, the third full attention layer can determine the target historical word set corresponding to the target linear attention layer based on the state update information related to the target linear attention layer, and transmit the target historical word set to the target linear attention layer. The state update information related here may include the state update strength information, key vector strength information, value vector strength information, gating information, or other information related to historical recursive state updates when the target linear attention layer processes the current word. By introducing the state update information related to the target linear attention layer itself, the third full attention layer can determine different target historical word sets for the state update requirements of different LAs.

[0174] In one embodiment of this application, when the third full attention layer uses the Top-K method to determine the target historical word set, the K value in the Top-K for different LAs can be the same or different. Here, the K value can be understood as the number of historical words selected into the target historical word set. In one implementation, the third full attention layer can use the same K value for multiple LAs. For example, the third full attention layer can select K historical words for each LA and use the corresponding K historical words as the target historical word set for that LA. This method is relatively simple to implement and facilitates the use of a unified selection rule among multiple LAs.

[0175] In another embodiment of this application, the third full attention layer can use different K values ​​for different attention layers (LAs). For example, for LAs with strong state update requirements and need to refer to more historical information, a larger K value can be set so that they receive a target historical word set containing more historical words; for LAs with weak state update requirements or need to control computational load, a smaller K value can be set so that they receive a smaller number of target historical words. The K value corresponding to different LAs can be determined based on the state update information of the corresponding LA, its position in the network layer, computational resource constraints, or task requirements. This application does not limit the K value corresponding to each LA to be the same.

[0176] It should be noted that different K values ​​corresponding to different LAs are not a necessary condition for this application. In some cases, even if the third full attention layer determines the target historical word set based on the state update information of different LAs, the number of target historical words corresponding to different LAs may be the same. In other cases, the number of target historical words corresponding to different LAs may be different. Therefore, the focus of this embodiment is that the third full attention layer can determine the corresponding target historical word set for the corresponding LA according to the state update requirements of different LAs, rather than simply using a fixed and completely identical target historical word selection result.

[0177] In some embodiments of this application, the third full attention layer can employ a sparse full attention mechanism to determine the target historical word set. For example, the third full attention layer can employ a full attention mechanism with Top-K sparse selection to select a subset of historical words as the target historical word set. By employing a sparse selection method, the third full attention layer does not need to perform full attention calculations on all historical words, which helps to control the additional computational overhead introduced by the full attention branch.

[0178] pass Figure 8 The second architecture shown features a third full-attention layer that can serve as an auxiliary selection branch, providing a target historical lexical set for one or more serial LAs. Compared to the approach of compressing and modeling historical information solely based on the LA's own recursive state, this embodiment introduces the FA's ability to select key historical lexicals, improving the utilization of key historical information when the LA updates its historical recursive state. Unlike a simple parallel fusion architecture, this embodiment does not simply fuse the FA output and the LA output directly; instead, it allows the target historical lexical set determined by the FA to directly participate in the LA's historical recursive state update process, thereby enabling a closer collaboration between the full-attention mechanism and the linear attention mechanism at the state update level.

[0179] When FA uses a sparse selection method Figure 8The architecture shown can provide fine-grained historical information selection results for multiple attention networks while controlling computational overhead. Therefore, this architecture improves the utilization of local key historical information by hybrid attention networks while retaining the low-overhead recursive modeling capabilities of attention networks.

[0180] Figure 9 This is a second schematic diagram of the second architecture of the hybrid attention network in this application. (See diagram below.) Figure 9 As shown, in one embodiment of this application, in Figure 8 Based on the illustrated embodiment, the hybrid attention network can be configured with multiple parallel auxiliary units along the network processing direction. Each parallel auxiliary unit may include multiple LAs connected in series and an FA configured in parallel with the multiple LAs. The FA is used to determine a corresponding target historical word set for at least one LA in the same parallel auxiliary unit, and transmit the target historical word set to the corresponding LA, so that the corresponding LA updates its historical recursion state according to the target historical word set.

[0181] exist Figure 9 In the illustrated embodiment, multiple parallel auxiliary units can be connected sequentially along the network processing direction. The output of the preceding parallel auxiliary unit can be transmitted to the following parallel auxiliary unit, enabling the hybrid attention network to utilize the FA (Factor-Aspect) to provide target historical word sets for the corresponding multiple LAs at different network stages. Thus, the hybrid attention network can introduce the historical word selection results of the full attention mechanism at multiple depth locations and use these selection results for the recursive state update of the corresponding linear attention layer.

[0182] It should be noted that, Figure 9 Only an example of two parallel auxiliary units connected in sequence is shown. In other embodiments, the number of parallel auxiliary units can be set according to model size, sequence length, computational resources, or task requirements. The number of LAs in each parallel auxiliary unit, the method by which the FA transmits the target historical word set to the LA, and the number of target historical words corresponding to each LA can all adopt the aforementioned... Figure 8 The relevant methods are set in the embodiments.

[0183] It should be noted that the aforementioned collaborative processing method involving full attention and linear attention can be applied to... Figures 6 to 9 In the hybrid attention network architecture shown. Figure 6 and Figure 7 The architecture shown primarily achieves the fusion of the output of the full attention layer and the output of the linear attention layer through a merging layer. Figure 8 and Figure 9The architecture shown primarily provides the target historical lexical set to one or more linear attention layers through parallel fully attention layers. Although the connection methods of the above architectures differ, they can all be used to achieve information exchange between the fully attention mechanism and the linear attention mechanism.

[0184] In some embodiments of this application, a full attention mechanism can be used to determine the target historical word set corresponding to the current word, and a linear attention mechanism can determine state decay information based on the target historical word set and update the historical recursive state based on the state decay information. Thus, the target historical word information selected by the full attention mechanism can be passed to the linear attention mechanism through a merging layer, bypass connection, or parallel auxiliary branch, enabling the linear attention mechanism to refer to key historical information related to the current word when updating the historical recursive state.

[0185] In some embodiments of this application, the target historical lexical set determined by the full attention mechanism can also be used to determine the target key vector, target value vector, and / or target query vector used by the linear attention mechanism during state update. In this way, when performing recursive state updates or attention output generation, the linear attention mechanism not only relies on the key, value, or query information corresponding to the current lexical itself, but can also further utilize the target historical lexical information filtered by the full attention mechanism.

[0186] In other embodiments of this application, the state update information of the linear attention mechanism corresponding to the current word can also be used to assist the full attention mechanism in determining the target historical word set. For example, when the full attention mechanism selects historical words, the relevance under different scoring dimensions can be weighted by combining the state update information of the linear attention mechanism, so that the target historical word set determined by the full attention mechanism better meets the state update requirements of the linear attention mechanism.

[0187] therefore, Figures 6 to 9 The architecture shown not only provides a network connection between the full attention mechanism and the linear attention mechanism, but also provides a structural foundation for the bidirectional collaboration between the aforementioned full attention mechanism and the linear attention mechanism. Compared with simply concatenating the full attention layer and the linear attention layer in sequence, or simply fusing the two types of attention outputs, the embodiments of this application enable the key historical word selection capability of the full attention mechanism and the recursive state modeling capability of the linear attention mechanism to be used in combination in multiple processing stages, thereby improving the utilization effect of key historical information in long sequence processing.

[0188] The following is a brief description of the technical scenarios in which the proposed solution can be applied: The data processing method based on hybrid attention proposed in this application can be applied to long sequence data processing scenarios. These scenarios may include long document understanding and generation, book content understanding and generation, long contextual dialogue, multi-turn task dialogue, long document retrieval and question answering, long code understanding and generation, long document translation, long video understanding and analysis, long audio understanding and recognition, and multimodal long sequence processing.

[0189] In the scenarios described above, the data to be processed typically includes lengthy contextual information or long historical sequence information. For example, in long document understanding and generation, long document retrieval and question answering, long document translation, or long code understanding and generation scenarios, the model needs to identify historical content related to the current processing position within a long text range; in long contextual dialogue or multi-turn task dialogue scenarios, the model needs to refer to multi-turn historical interaction content to generate output that matches the current dialogue state; in long video understanding and analysis scenarios, the model needs to obtain information related to the current analysis object from video frames, video segments, or spatiotemporal features over a long time range; in long audio understanding and recognition scenarios, the model needs to retain historical acoustic or semantic information related to the current audio segment from a long audio sequence.

[0190] When employing the scheme of this application embodiment, the full attention mechanism can determine the target historical word set related to the current word from a long historical sequence, and the linear attention mechanism can update the historical recursion state based on the relevance information corresponding to the target historical word set. Therefore, in long sequence scenarios, the linear attention mechanism can more effectively retain historical information related to the current word while maintaining low computational and storage overhead, and reduce the continued impact of less relevant historical information on the current recursion state.

[0191] Furthermore, in cloud-based inference scenarios, the embodiments of this application can reduce the resource pressure caused by the storage of historical key-value information and attention calculation during long context processing, which is beneficial to improving the resource utilization efficiency of long sequence inference. In edge device or end-device inference scenarios, since device computing and storage resources are usually limited, the embodiments of this application compress and model historical information through a linear attention mechanism, and combine the target historical lexical selection results provided by the full attention mechanism, which can improve the model's utilization of key historical information while controlling the amount of computation and storage.

[0192] Therefore, the embodiments of this application are applicable to application scenarios that require processing long contexts, long time spans, or cross-modal historical information. For the above scenarios, the embodiments of this application can improve the utilization effect of key historical information in the state update, information reading, and output generation processes without excessively increasing the processing cost of long sequences, thereby improving the problem of insufficient retention of historical information or persistent residual irrelevant historical information during the processing of long sequence data.

[0193] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0194] Based on the same concept, this application also provides a data processing apparatus based on hybrid attention, which can be used to implement the data processing method based on hybrid attention described in the above embodiments, as described in the following embodiments. Since the principle of the data processing apparatus based on hybrid attention is similar to that of the data processing method based on hybrid attention, embodiments of the data processing apparatus based on hybrid attention can be found in embodiments of the data processing method based on hybrid attention, and repeated descriptions will not be repeated. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0195] Figure 10 This is a structural block diagram of a data processing device based on hybrid attention, as described in an embodiment of this application. Figure 10 As shown, in one embodiment of this application, the data processing apparatus based on hybrid attention includes: The target historical word set determination unit 1 is used to determine the target historical word set corresponding to the current word in the word sequence corresponding to the data to be processed by using the full attention mechanism in the hybrid attention network when processing the current word. The state decay information determination unit 2 is used to determine the state decay information of the linear attention mechanism in the hybrid attention network based on the correlation between the current word and the target historical word in the target historical word set. The recursive state update unit 3 is used to update the historical recursive state of the linear attention mechanism according to the state decay information, so as to obtain the current recursive state of the linear attention mechanism corresponding to the current word.

[0196] In one embodiment of this application, the data processing apparatus based on hybrid attention further includes: The target vector determination unit is used to determine the target key vector and target value vector corresponding to the current word in the linear attention mechanism based on the key vector and value vector corresponding to the target historical word in the target historical word set; The recursive state update unit 3 is specifically used to update the historical recursive state based on the state decay information, the target key vector, and the target value vector.

[0197] In one embodiment of this application, the target historical lexical set determination unit 1 includes: The dimension relevance determination module is used to determine the dimension relevance between the current word element and each historical word element in the rating dimension for each of the multiple rating dimensions, based on the query vector of the current word element in the rating dimension and the key vectors corresponding to each historical word element. The weight coefficient determination module is used to determine the weight coefficient corresponding to the scoring dimension based on the state update information of the current word element corresponding to the linear attention mechanism. The selection score determination module is used to weight the dimensional relevance of the same historical word element under the multiple scoring dimensions according to the weight coefficients corresponding to the multiple scoring dimensions, so as to obtain the selection score corresponding to the historical word element. The target historical word set determination module is used to determine the target historical word set based on the selection score corresponding to each historical word.

[0198] In one embodiment of this application, the recursive state update unit 3 includes: The state update information determination module is used to determine the state update information of the linear attention mechanism based on the query vector and value vector corresponding to the current word element. The recursive state update module is used to update the historical recursive state based on the state decay information and the state update information to obtain the current recursive state.

[0199] In one embodiment of this application, the target vector determination unit includes: The selected historical word element determination module is used to determine the selected historical word element from the target historical word element set based on the correlation between each target historical word element in the target historical word element set and the current word element; The corresponding vector determination module is used to determine the key vector and value vector corresponding to the selected historical word as the target key vector and the target value vector.

[0200] In one embodiment of this application, the data processing apparatus based on hybrid attention further includes: The target query vector determination unit is used to determine the target query vector corresponding to the current word in the linear attention mechanism based on the query vector corresponding to the target historical word in the target historical word set. The attention output generation unit is used to generate the attention output of the linear attention mechanism corresponding to the current word based on the current recursive state and the target query vector.

[0201] In one embodiment of this application, the weight coefficient determination module is specifically used to determine the weight coefficient corresponding to the scoring dimension based on the state update strength information of the current word corresponding to the linear attention mechanism, the value vector strength information of the current word under the scoring dimension, and the key vector strength information corresponding to the scoring dimension.

[0202] To achieve the above objectives, according to another aspect of this application, a computer device is also provided. For example... Figure 11 As shown, the computer device includes a memory, a processor, a communication interface, and a communication bus. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements the steps in the method of the above embodiments.

[0203] The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0204] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and units, such as the program units corresponding to those in the above method embodiments of this application. The processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the methods in the above method embodiments.

[0205] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0206] The one or more units are stored in the memory and, when executed by the processor, perform the methods described in the above embodiments.

[0207] The specific details of the aforementioned computer equipment can be understood by referring to the relevant descriptions and effects in the above embodiments, and will not be repeated here.

[0208] To achieve the above objectives, according to another aspect of this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed in a computer processor, implements the steps in the above-described data processing method based on hybrid attention. Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium may also include combinations of the above types of memory.

[0209] To achieve the above objectives, according to another aspect of this application, a computer program product is also provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method based on hybrid attention.

[0210] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device, or fabricating them separately as individual integrated circuit modules, or fabricating multiple modules or steps as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0211] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A data processing method based on hybrid attention, characterized in that, include: When processing the current word in the word sequence corresponding to the data to be processed, the target historical word set corresponding to the current word is determined by the full attention mechanism in the hybrid attention network; Based on the correlation between the current word and the target historical word in the target historical word set, the state decay information of the linear attention mechanism in the hybrid attention network is determined; Based on the state decay information, the historical recursive state of the linear attention mechanism is updated to obtain the current recursive state of the linear attention mechanism corresponding to the current word.

2. The data processing method based on hybrid attention according to claim 1, characterized in that, Also includes: Based on the key vector and value vector corresponding to the target historical word in the target historical word set, the target key vector and target value vector corresponding to the current word in the linear attention mechanism are determined. The step of updating the historical recursive state of the linear attention mechanism based on the state decay information includes: The historical recursive state is updated based on the state decay information, the target key vector, and the target value vector.

3. The data processing method based on hybrid attention according to claim 1, characterized in that, The step of determining the target historical word set corresponding to the current word element through the full attention mechanism in a hybrid attention network includes: For each of the multiple rating dimensions, the dimensional correlation between the current word element and each historical word element in that rating dimension is determined based on the query vector of the current word element in that rating dimension and the key vectors corresponding to each historical word element. Based on the state update information of the current word element corresponding to the linear attention mechanism, determine the weight coefficient corresponding to this scoring dimension; Based on the weight coefficients corresponding to the multiple scoring dimensions, the dimensional relevance of the same historical word element under the multiple scoring dimensions is weighted to obtain the selection score corresponding to the historical word element. The target set of historical lexical units is determined based on the selection scores corresponding to each historical lexical unit.

4. The data processing method based on hybrid attention according to claim 1, characterized in that, The hybrid attention network includes a first full attention layer, n linear attention layers, and a second full attention layer connected in sequence, where n is an integer greater than or equal to 1; At least one of the n linear attention layers is followed by a merging layer; The attention output of the first full attention layer is transmitted to the adjacent linear attention layer and the merging layer; The merging layer merges the attention output of the first full attention layer and the attention output of the corresponding linear attention layer to obtain a merged output, and then transmits the merged output to the next network layer.

5. The data processing method based on hybrid attention according to claim 1, characterized in that, The hybrid attention network includes m linear attention layers connected in sequence and a third full attention layer connected in parallel with the m linear attention layers, where m is an integer greater than or equal to 1; The third full attention layer is used to determine the target historical word set corresponding to the current word, and to transmit the target historical word set to at least one of the m linear attention layers; The linear attention layer that receives the target historical word set updates the historical recursive state based on the target historical word set.

6. The data processing method based on hybrid attention according to claim 1, characterized in that, The step of updating the historical recursive state of the linear attention mechanism based on the state decay information to obtain the current recursive state of the linear attention mechanism corresponding to the current word includes: Based on the query vector and value vector corresponding to the current word, determine the state update information of the linear attention mechanism; Based on the state decay information and the state update information, the historical recursive state is updated to obtain the current recursive state.

7. The data processing method based on hybrid attention according to claim 2, characterized in that, The step of determining the target key vector and target value vector corresponding to the current word in the linear attention mechanism based on the key vector and value vector corresponding to the target historical word in the target historical word set includes: Based on the correlation between each target historical word in the target historical word set and the current word, the selected historical word is determined from the target historical word set; The key vector and value vector corresponding to the selected historical word are determined as the target key vector and the target value vector.

8. The data processing method based on hybrid attention according to claim 2, characterized in that, Also includes: Based on the query vector corresponding to the target historical word in the target historical word set, the target query vector corresponding to the current word in the linear attention mechanism is determined; Based on the current recursive state and the target query vector, the attention output corresponding to the current word in the linear attention mechanism is generated.

9. The data processing method based on hybrid attention according to claim 3, characterized in that, The step of determining the weight coefficients corresponding to the scoring dimension based on the state update information of the current word unit corresponding to the linear attention mechanism includes: Based on the state update strength information of the current word corresponding to the linear attention mechanism, the value vector strength information of the current word under the scoring dimension, and the key vector strength information corresponding to the scoring dimension, the weight coefficient corresponding to the scoring dimension is determined.

10. The data processing method based on hybrid attention according to claim 4, characterized in that, Each of the n linear attention layers is followed by the merging layer; The attention output of the first full attention layer is transmitted to each of the merging layers, and each merging layer merges the attention output of the first full attention layer and the attention output of the corresponding linear attention layer.

11. The data processing method based on hybrid attention according to claim 5, characterized in that, The third full attention layer transmits the corresponding target historical word set to each of the m linear attention layers.

12. The data processing method based on hybrid attention according to claim 5, characterized in that, For the target linear attention layer among the m linear attention layers, the third full attention layer determines the target historical word set corresponding to the target linear attention layer based on the state update information corresponding to the target linear attention layer, and transmits the corresponding target historical word set to the target linear attention layer.

13. A data processing device based on hybrid attention, characterized in that, include: The target historical word set determination unit is used to determine the target historical word set corresponding to the current word in the word sequence corresponding to the data to be processed by using the full attention mechanism in the hybrid attention network. The state decay information determination unit is used to determine the state decay information of the linear attention mechanism in the hybrid attention network based on the correlation between the current word and the target historical word in the target historical word set. The recursive state update unit is used to update the historical recursive state of the linear attention mechanism according to the state decay information, so as to obtain the current recursive state of the linear attention mechanism corresponding to the current word.

14. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 12.

15. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 12.

16. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 12.

Images