Linear attention mechanism methods, devices and electronic devices for vision tasks

Abstract

This invention relates to the field of artificial intelligence technology and discloses a linear attention mechanism method, device, and electronic device for visual tasks. The method includes: performing norm decomposition and direction decomposition on the input query vector and key vector respectively to obtain the magnitude and unit direction vector corresponding to the query vector and key vector; designing a query norm-driven dynamic entropy reduction control mechanism and a cosine suppression nonnegativity preservation mechanism to obtain the query kernel function feature vector and the key kernel function feature vector; and finally calculating the linear attention result based on the query kernel function feature vector and the key kernel function feature vector. This invention significantly improves the model's ability to process information of different intensities while maintaining linear computational complexity, enhances the model's performance in complex tasks, and thus achieves a balance between efficiency and expressive power, effectively solving the problem of the lack of correlation between norm and entropy value in traditional linear attention mechanisms.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically to linear attention mechanism methods, devices, and electronic devices for visual tasks. Background Technology

[0002] Standard self-attention mechanisms model long-distance dependencies by calculating the interaction weights between all pairs of elements in a sequence. However, the core matrix operations cause the computational complexity to increase quadratically with the sequence length, resulting in excessive computational resource consumption and high inference latency when processing high-resolution images or long video sequences.

[0003] To overcome this efficiency bottleneck, related technologies have proposed linear attention mechanisms, aiming to reduce complexity to a linear level through kernel function approximation. However, existing linear attention methods suffer from accuracy degradation when approximating the dynamic characteristics of standard attention, mainly due to two technical defects: first, the kernel function design cannot perceive the magnitude information of the query vector, resulting in a disconnect between the concentration of attention distribution and the input intensity; second, to ensure the non-negativity of the kernel function, negative vector components are over-filtered, losing crucial semantic antonymous interaction information. Summary of the Invention

[0004] This invention provides a linear attention mechanism method, device, and electronic device for visual tasks, to solve the problem that existing linear attention mechanisms cannot simultaneously achieve dynamic control of attention entropy value by query norm decomposition and effective preservation of negative vector components, resulting in their expressive power failing to reach the standard attention level.

[0005] In a first aspect, the present invention provides a linear attention mechanism method for visual tasks, the method comprising:

[0006] Perform norm decomposition and direction decomposition on the input query vector and key vector respectively to obtain the magnitude and unit direction vector corresponding to the query vector and key vector;

[0007] Based on the magnitude of the query vector, a magnitude-aware kernel function is used to dynamically modulate the query unit direction vector, and a fixed magnitude modulation is performed on the key unit direction vector based on the magnitude of the key vector.

[0008] Map the query unit direction vector and the key unit direction vector to trigonometric function features to obtain the query direction feature vector and the key direction feature vector, respectively.

[0009] The modulated query amplitude information is fused with the query direction vector feature to obtain the query kernel function feature vector; the modulated key amplitude information is fused with the key direction feature to obtain the key kernel function feature vector.

[0010] The linear attention result is calculated based on the query kernel function feature vector and the key kernel function feature vector.

[0011] The linear attention mechanism method for visual tasks provided by this invention adaptively adjusts the concentration of attention distribution according to the query vector norm decomposition through modulus-aware dynamic amplitude modulation, accurately reproducing the entropy change characteristics of standard attention. Simultaneously, it employs a combination of trigonometric function mapping and inner product calculation to fully preserve vector direction interaction information while maintaining non-negativity, avoiding semantic degradation caused by the loss of negative components. This mechanism significantly improves the model's ability to focus on key information and the accuracy of semantic relationship modeling while maintaining linear computational complexity, thus achieving a balance between efficiency and expressive power. It solves the problem that existing linear attention mechanisms cannot simultaneously achieve dynamic control of attention entropy values ​​by query norm decomposition and effective preservation of negative vector components, resulting in their expressive power failing to reach the level of standard attention.

[0012] In one optional implementation, norm decomposition and direction decomposition are performed on the input query vector and key vector, respectively, to obtain the magnitude and unit direction vector corresponding to the query vector and key vector, including:

[0013] Calculate the magnitude of the query vector, and based on the query vector and its corresponding magnitude, calculate the query unit direction vector;

[0014] Calculate the magnitude of the bond vector, and based on the bond vector and its corresponding magnitude, calculate the bond unit direction vector.

[0015] The linear attention mechanism method for visual tasks provided by this invention achieves decoupling and separation of vector intensity information and direction information by calculating the magnitude length and unit direction vector of the query vector and key vector, providing a clear foundation for subsequent independent adjustment of amplitude characteristics and direction interaction; at the same time, this decomposition process only involves basic mathematical operations, which is computationally efficient and does not lose the original information, ensuring the accuracy and efficiency of subsequent processing.

[0016] In one optional implementation, based on the magnitude of the query vector, a magnitude-aware kernel function is used to dynamically modulate the magnitude of the query unit direction vector, including:

[0017] Based on the magnitude of the query vector, the dynamic modulation power is calculated using the magnitude-aware kernel function;

[0018] According to the dynamic modulation power, each component of the query unit direction vector is subjected to exponentiation, and the components after exponentiation are combined to obtain the query amplitude modulation vector.

[0019] The linear attention mechanism method for visual tasks provided by this invention generates a dynamic power by nonlinearly generating the query modulus, so that the amplitude modulation intensity is adapted to the original intensity of the query in real time, thereby achieving fine-grained control of the degree of attention concentration. The power operation is performed independently on each component of the unit direction vector, which preserves the integrity and local differences of the direction information, thereby enhancing the query focusing ability while avoiding the control mismatch problem caused by the fixed power.

[0020] In one optional implementation, the unit direction vector of the bond is modulated with a fixed amplitude based on the magnitude of the bond vector, including:

[0021] The magnitude of the key vector is used as the input for fixed amplitude modulation. A preset fixed power is used to exponentiate the magnitude of the key vector, and the result of the power operation is used as the key amplitude scalar.

[0022] The linear attention mechanism method for visual tasks provided by this invention uses a preset fixed power to perform power operations on the magnitude of the key vector to generate a key amplitude scalar, providing a stable amplitude basis for key direction features in a simple and efficient manner. The fixed power design avoids the computational overhead and parameter dependence caused by dynamic adjustment, so that the amplitude information of the key can be guaranteed to be numerically stable while complementing the dynamic modulation of the query, jointly supporting the accurate calculation of subsequent direction interactions.

[0023] In one optional implementation, the query unit direction vector and the key unit direction vector are mapped to trigonometric function features, respectively, to obtain the query direction feature vector and the key direction feature vector, including:

[0024] Obtain each component of the query unit direction vector, calculate the cosine and sine values ​​respectively, and arrange all the calculated cosine values ​​in order to form the query cosine branch vector, and arrange all the calculated sine values ​​in order to form the query sine branch vector;

[0025] The query cosine branch vector and the query sine branch vector are concatenated to obtain the query direction feature vector;

[0026] Obtain each component of the bond unit direction vector, calculate the cosine and sine values ​​respectively, and arrange all the calculated cosine values ​​in order to form the bond cosine branch vector, and arrange all the calculated sine values ​​in order to form the bond sine branch vector;

[0027] By concatenating the bond cosine branch vector with the bond sine branch vector, we obtain the bond direction feature vector.

[0028] The linear attention mechanism method for visual tasks provided by this invention constructs and concatenates cosine and sine branches by calculating the cosine and sine values ​​of each component of the unit direction vector, thus completely mapping the direction information of the query and key into a trigonometric function feature vector. This mapping preserves the polarity differences of all components, avoids information loss caused by filtering negative values ​​in traditional activation functions, and utilizes the inherent properties of trigonometric functions to achieve a continuous and bounded representation of direction features, providing a precise foundation for subsequent attention calculations based on directional interaction.

[0029] In one optional implementation, the modulated query amplitude information is fused with the query direction vector features to obtain a query kernel function feature vector, including:

[0030] Multiply the query amplitude modulation vector and the query cosine branch vector element by element to obtain the modulated query cosine branch;

[0031] Multiply the query amplitude modulation vector and the query sine branch vector element by element to obtain the modulated query sine branch;

[0032] The modulated query cosine branch and the modulated query sine branch are concatenated to obtain the query kernel function feature vector.

[0033] The linear attention mechanism method for visual tasks provided by this invention achieves refined fusion of amplitude and direction information by multiplying the query amplitude modulation vector element-wise with the query cosine and sine branches respectively. The element-wise multiplication preserves the local characteristics of each direction component, so that the modulated cosine and sine branches simultaneously carry the intensity information of the magnitude perception and the original polarity difference. The two modulated branches are concatenated to form the query kernel function feature vector, which provides a unified representation for subsequent direction interaction calculation that simultaneously contains the amplitude modulation result and the complete direction feature, ensuring the integrity of information transmission and the efficiency of computation.

[0034] In one optional implementation, the modulated bond amplitude information is fused with the bond orientation features to obtain a bond kernel function feature vector, including:

[0035] The modulated key cosine branch is obtained by multiplying the key amplitude scalar with the key cosine branch vector element by element.

[0036] The modulated key sinusoidal branch is obtained by multiplying the key amplitude scalar with the key sinusoidal branch vector element by element.

[0037] The modulated key cosine branch and the modulated key sine branch are concatenated to obtain the key kernel function feature vector.

[0038] The linear attention mechanism method for visual tasks provided by this invention achieves the fusion of key amplitude information and directional features by performing element-wise multiplication of the key amplitude scalar with the key cosine branch and the key sine branch respectively. The element-wise multiplication ensures that the modulation of the amplitude scalar on each directional component remains consistent, while fully preserving the polarity difference information carried by the cosine and sine branches. The two modulated branches are concatenated to form the key kernel function feature vector, providing a unified representation for subsequent attention calculation that simultaneously contains a stable amplitude basis and complete directional features, ensuring information integrity and computational consistency when interacting with query features.

[0039] In one optional implementation, a linear attention result is calculated based on the query kernel function feature vector and the key kernel function feature vector, including:

[0040] Calculate the matrix multiplication of the query kernel function feature vector with the transpose of the feature vectors of all key kernel functions to obtain the original attention score matrix;

[0041] Calculate the matrix multiplication of the query kernel function eigenvector and the transpose of the eigenvectors of all key kernel functions to obtain the normalized denominator vector;

[0042] The original attention score matrix is ​​divided position by position by the normalized denominator vector to obtain the normalized attention weight matrix.

[0043] The normalized attention weight matrix is ​​multiplied by the pre-calculated value vector matrix to obtain the weighted sum of the output feature matrix, which is then output as the result of the linear attention calculation.

[0044] The linear attention mechanism method for visual tasks provided by this invention calculates the attention score and normalized denominator through two matrix multiplications, normalizes the weights using division, and finally obtains the output features by weighted summation with the value vector matrix. The entire process adopts matrix operation implementation, which completes the mapping from kernel function features to the final attention result while maintaining linear complexity. This ensures that the preceding amplitude modulation and direction interaction information can be accurately transmitted to the output and achieves effective aggregation of value vectors.

[0045] In a second aspect, the present invention provides a linear attention mechanism device for visual tasks, the device comprising:

[0046] The norm and direction decomposition module is used to perform norm decomposition and direction decomposition on the input query vector and key vector respectively, to obtain the magnitude and unit direction vector of the query vector and key vector;

[0047] The module with modulus-aware kernel function is used to dynamically modulate the query unit direction vector based on the modulus of the query vector, and to perform fixed amplitude modulation on the key unit direction vector based on the modulus of the key vector.

[0048] The direction mapping module is used to map the query unit direction vector and the key unit direction vector to trigonometric function features, respectively, to obtain the query direction feature vector and the key direction feature vector.

[0049] The direction interaction calculation module is used to fuse the modulated query amplitude information with the query direction vector features to obtain the query kernel function feature vector; and to fuse the modulated key amplitude information with the key direction features to obtain the key kernel function feature vector.

[0050] The linear attention output module is used to calculate the linear attention result based on the query kernel function feature vector and the key kernel function feature vector.

[0051] Thirdly, the present invention provides an electronic device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the linear attention mechanism method for vision tasks described in the first aspect or any corresponding embodiment thereof.

[0052] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the linear attention mechanism method for vision tasks described in the first aspect or any corresponding embodiment thereof.

[0053] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the linear attention mechanism method for vision tasks described in the first aspect or any corresponding embodiment thereof. Attached Figure Description

[0054] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0055] Figure 1 This is a schematic diagram of an application scenario according to an embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram of a first method for a linear attention mechanism for visual tasks according to an embodiment of the present invention.

[0057] Figure 3 This is a second flowchart illustrating a linear attention mechanism method for visual tasks according to an embodiment of the present invention.

[0058] Figure 4 This is a schematic diagram of the third process of a linear attention mechanism method for visual tasks according to an embodiment of the present invention.

[0059] Figure 5 This is a schematic diagram illustrating the dynamic action of the modulus sensing kernel function according to an embodiment of the present invention;

[0060] Figure 6 This is a schematic diagram of the nonnegativity preservation process of cosine suppression according to an embodiment of the present invention;

[0061] Figure 7 This is a schematic diagram of the fourth process of a linear attention mechanism method for visual tasks according to an embodiment of the present invention;

[0062] Figure 8 This is a structural block diagram of a linear attention mechanism device for visual tasks according to an embodiment of the present invention;

[0063] Figure 9 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in the present invention and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.

[0066] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified. As an optional application scenario of an embodiment of this invention, such as... Figure 1As shown, application 101 is installed in terminal device 110, and user 130 can interact with application 101 through terminal device 110 and / or access device of terminal device 110.

[0067] For example, application 101 can be any application that provides question-and-answer related services. For instance, application 101 could be a question-and-answer interactive application, such as a text-to-text application, an image-to-text application, etc. Figure 1 In the application scenario shown, if application 101 is active, the terminal device 110 can display the interface 102 of application 101. The interface 102 may include various pages that application 101 can provide, such as interactive pages, settings pages, query pages, etc.

[0068] In some embodiments, terminal device 110 is communicatively connected to server 120 to provide services to application 101. Terminal device 110 may be a mobile terminal, fixed terminal, or portable terminal, etc., including but not limited to mobile phones, desktop computers, laptop computers, multimedia tablets, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, terminal device 110 may also support any type of interface, and server 120 may be various types of computing systems or servers capable of providing computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, etc.

[0069] It should be noted that, Figure 1 This is merely an example of an application scenario and does not limit the scope of protection of this invention.

[0070] The embodiments of the present invention will now be described with reference to the accompanying drawings. It should be understood that the pages shown in the drawings are merely examples, and various page designs are possible in practice. The various graphic elements on the page may have different arrangements and different visual representations; one or more elements may be omitted or replaced, and one or more other elements may also be present, without any limitation in the embodiments of the present invention. Furthermore, the embodiments described below primarily pertain to terminal device 110. It should be understood that the actions described relative to terminal device 110 can be performed by application 101 on terminal device 110, or can be performed by application 101 in conjunction with its server (e.g., server 120).

[0071] According to an embodiment of the present invention, a linear attention mechanism method for visual tasks is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0072] This embodiment provides a linear attention mechanism method for visual tasks, which can be used in the aforementioned electronic devices or terminal devices. Figure 2 This is a flowchart of a linear attention mechanism method for vision tasks according to an embodiment of the present invention, as shown below. Figure 2 As shown, the process includes the following steps:

[0073] Step S201: Perform norm decomposition and direction decomposition on the input query vector and key vector respectively to obtain the magnitude and unit direction vector corresponding to the query vector and key vector.

[0074] The query vector represents the current focus for finding relevant information, while the key vector represents the content of all candidate locations for matching. The attention weight is determined by similarity calculation between the two.

[0075] Norm decomposition calculates the magnitude of a vector, representing its overall strength or amplitude. Direction decomposition divides a vector by its magnitude to obtain a unit direction vector, representing the direction of the vector or the relative proportions of its components. First, the magnitudes of the query vector and key vector are calculated separately to obtain their strength information. Then, the query vector and key vector are divided by their corresponding magnitudes to obtain unit direction vectors that retain only their directional characteristics. This process decouples vector strength from direction information, providing a foundation for subsequent independent control of amplitude characteristics and direction interaction.

[0076] Step S202: Based on the magnitude of the query vector, a magnitude-aware kernel function is used to dynamically modulate the query unit direction vector, and a fixed magnitude modulation is performed on the key unit direction vector based on the magnitude of the key vector.

[0077] Dynamic amplitude modulation refers to adjusting the power of the kernel function in real time according to the magnitude of the query vector, and performing power operations on each component of the query unit direction vector to dynamically adapt the modulation intensity to the original intensity of the query, thereby achieving fine-grained control over the degree of attention concentration.

[0078] Fixed amplitude modulation refers to using a preset fixed power to perform power operations on the magnitude of the key vector to generate a scalar as the basis for the key's amplitude, providing stable amplitude information for key direction features that complements the dynamic adjustment of the query function.

[0079] Step S202 includes the design process of the modulus-aware kernel function. In this embodiment, the modulus-aware kernel function is key to implementing the query norm-driven dynamic entropy reduction control mechanism. Traditional linear attention mechanisms' kernel functions cannot sense changes in the modulus of the query vector, resulting in the entropy value of the attention distribution not being effectively adjusted according to the query intensity, leading to shortcomings in the model's ability to focus on key information. To overcome this limitation, this embodiment proposes a novel method for constructing the modulus-aware kernel function, namely, designing a query norm-driven dynamic entropy reduction control mechanism, specifically including:

[0080] Construct a norm-adaptive kernel function, and define the exponentiation of the query kernel function as... This causes the power to grow non-linearly with the query norm, i.e., dynamic amplitude modulation of the query unit direction vector based on the magnitude of the query vector. Wherein, To query the dynamic exponentiation (scalar) of the kernel function. The magnitude (scalar) of the query vector represents the strength or amplitude information of the query vector and serves as the input basis for dynamic control. These are global hyperparameters used to control the exponentiation. The overall range and magnitude of change, and the sensitivity to adjust attention entropy; The hyperbolic tangent function is a nonlinear activation function that maps the input to the interval (-1, 1) to achieve nonlinear growth of the power with the modulus and ensure numerical stability.

[0081] Through mathematical derivation, it is proven that when both the derivative of the kernel function and the second derivative are positive, the positive sequence entropy of the attention distribution decreases strictly and monotonically with the query norm, accurately reproducing the dynamic focusing characteristics of the softmax function and overcoming the defect that the gating parameters of Gated LinearAttention (GLA) are independent of the norm.

[0082] Step S203: Map the query unit direction vector and the key unit direction vector to trigonometric function features respectively to obtain the query direction feature vector and the key direction feature vector.

[0083] Specifically, trigonometric function characteristics refer to the characteristic representation obtained by transforming a unit direction vector using cosine and sine functions.

[0084] This embodiment provides a non-negativity preservation mechanism for cosine suppression, which maps the unit direction vector to a trigonometric function. It also uses the cosine difference formula to transform the interaction between the query and the key direction into a non-negative value. This mechanism only suppresses the reverse component while retaining the enhancement effect of the same-direction component. It avoids the information loss problem caused by the ReLU (Rectified Linear Unit) activation function filtering negative values, as well as the shortcomings of PolaFormer (polarity-aware linear attention mechanism) in blurring polarity differences by adding positive and negative components.

[0085] Step S204: The modulated query amplitude information is fused with the query direction vector feature to obtain the query kernel function feature vector; the modulated key amplitude information is fused with the key direction feature to obtain the key kernel function feature vector.

[0086] Among them, the query kernel function feature vector is a vector obtained by fusing the amplitude modulation result of the query with the trigonometric function direction feature of the query, and is used to characterize the intensity and direction information of the query; the key kernel function feature vector is a vector obtained by fusing the fixed amplitude scalar of the key with the trigonometric function direction feature of the key, and is used to characterize the intensity and direction information of the key. The two interact in direction through inner product calculation.

[0087] Step S205: Based on the query kernel function feature vector and the key kernel function feature vector, calculate the linear attention result.

[0088] Specifically, the linear attention result refers to the weighted aggregated feature vector output after reducing the computational complexity of traditional softmax attention from quadratic to linear through kernel function mapping. It is essentially a weighted sum of value vectors, where the weights are calculated by the inner product of the query kernel feature vector and the key kernel feature vector. This achieves dynamic focusing and semantic interaction of input information while maintaining linear complexity.

[0089] This embodiment presents a linear attention mechanism method for vision tasks, constructing and organically integrating a modulus-aware linear attention architecture. The query norm-driven dynamic entropy reduction control mechanism and the non-negativity preservation mechanism of cosine suppression are integrated into the Transformer module, replacing the traditional attention layer. This forms an end-to-end process encompassing input normalization projection, norm-direction decomposition, kernel function mapping, and linear attention computation. This architecture can directly replace existing modules without modifying the overall Transformer framework, exhibiting excellent compatibility across multiple task scenarios.

[0090] This embodiment provides a linear attention mechanism method for visual tasks, which can be used in the aforementioned electronic devices or terminal devices. Figure 3 This is a flowchart of a linear attention mechanism method for vision tasks according to an embodiment of the present invention, as shown below. Figure 3 As shown, the process includes the following steps:

[0091] Step S301: Perform norm decomposition and direction decomposition on the input query vector and key vector respectively to obtain the magnitude and unit direction vector corresponding to the query vector and key vector.

[0092] Specifically, step S301 includes:

[0093] Step S3011: Calculate the magnitude of the query vector, and calculate the query unit direction vector based on the query vector and its corresponding magnitude.

[0094] Specifically, firstly, the square root of the sum of squares of each component of the query vector is performed to obtain the query modulus representing the overall strength of the vector; then, each component of the original query vector is divided by this modulus to obtain a unit direction vector with a length normalized to 1 but retaining the original direction. This process decouples the query vector strength information from the direction information, providing a foundation for subsequent independent control of amplitude characteristics and direction interaction.

[0095] Step S3012: Calculate the magnitude of the bond vector, and based on the bond vector and its corresponding magnitude, calculate the bond unit direction vector.

[0096] Specifically, the sum of squares and square root of each component of the bond vector are first performed to obtain the bond modulus length, which represents the overall strength of the vector. Then, each component of the original bond vector is divided by this modulus length to obtain a bond unit direction vector with a length normalized to 1 but retaining the original direction. This process decouples the bond vector strength information from the direction information, providing a foundation for subsequent independent control of amplitude characteristics and direction interaction.

[0097] Step S302: Based on the magnitude of the query vector, a magnitude-aware kernel function is used to dynamically modulate the query unit direction vector, and a fixed magnitude modulation is performed on the key unit direction vector based on the magnitude of the key vector.

[0098] Specifically, step S302 includes:

[0099] Step S3021: Based on the magnitude of the query vector, the dynamic modulation power is calculated using the magnitude-aware kernel function.

[0100] Specifically, the modulus-aware kernel function primarily operates on the query vector and the key vector. For the query vector, this embodiment defines an adaptive power function query kernel function.

[0101] Specifically, the power of the query kernel function is set to... Here, λ is a global hyperparameter used to control the overall range of entropy variation. In this design, the power of the query vector is not a fixed value, but rather varies with the magnitude of the query vector. It adjusts dynamically according to changes in the magnitude of the query vector. When it increases, The value of increases, thus making Non-linear growth. This means that the kernel function of the query vector... For querying the unit direction vector The amplification effect will be enhanced.

[0102] From a mathematical perspective, this embodiment rigorously proves through derivation that when the query kernel function satisfies the derivative... And the second derivative When the query norm increases, the positive sequence entropy (PSE) of the attention distribution decreases strictly and monotonically, which is highly consistent with the entropy change of softmax attention.

[0103] Step S3022: According to the dynamic modulation power, perform power operation on each component of the query unit direction vector, and combine the components after power operation to obtain the query amplitude modulation vector.

[0104] Specifically, firstly, the dynamic modulation exponent is used as the exponent to independently calculate the power value of each component of the query unit direction vector, so that each direction component obtains different degrees of amplitude scaling according to the original intensity of the query; then, all the components after the exponentiation are recombined in the original order to form a query amplitude modulation vector with the same dimension as the query unit direction vector. This process realizes the fine-grained control of the direction components by the query intensity information, so that the modulated amplitude vector simultaneously carries the original modulus characteristics and direction distribution characteristics of the query, providing a basis for intensity adaptation for subsequent fusion with trigonometric function direction characteristics.

[0105] Step S3023: The magnitude of the key vector is used as the input of fixed amplitude modulation. A preset fixed power is used to perform a power operation on the magnitude of the key vector, and the result of the power operation is used as the key amplitude scalar.

[0106] For key vectors, a fixed power function is used, i.e. Specifically, the magnitude of the key vector is first extracted as the numerical basis for characterizing the overall strength of the key vector. Then, the magnitude is calculated by exponentiation using a preset fixed power, which scales the amplitude information of the key according to a fixed nonlinear scale. Finally, the value obtained by the exponentiation is output as a scalar of the key amplitude. This scalar is fused with the trigonometric function direction features of the key to provide a stable amplitude basis for the key direction features that complements the dynamic modulation of the query, ensuring that the key has a consistent strength representation in the direction interaction calculation.

[0107] Step S303: Map the query unit direction vector and the key unit direction vector to trigonometric function features, respectively, to obtain the query direction feature vector and the key direction feature vector. For details, please refer to [link to relevant documentation]. Figure 2Step S203 of the illustrated embodiment will not be described again here.

[0108] Step S304: Fuse the modulated query amplitude information with the query direction vector feature to obtain the query kernel function feature vector; fuse the modulated key amplitude information with the key direction feature to obtain the key kernel function feature vector. For details, please refer to [link to relevant documentation]. Figure 2 Step S204 of the illustrated embodiment will not be described again here.

[0109] Step S305: Based on the query kernel function feature vector and the key kernel function feature vector, calculate the linear attention result. See details below. Figure 2 Step S205 of the illustrated embodiment will not be described again here.

[0110] The linear attention mechanism for visual tasks provided in this embodiment utilizes a modulus-aware kernel design, enabling the model to intelligently adjust the level of attention based on the modulus of the query vector when processing input. When the query vector modulus is large, attention is more concentrated, allowing the model to focus on key information; conversely, when the query vector modulus is small, attention is more dispersed, allowing the model to consider more background information. This dynamic adjustment mechanism effectively solves the problem of the lack of correlation between norm and entropy in traditional linear attention mechanisms, significantly improving the model's ability to process information of varying intensities and enhancing its performance in complex tasks.

[0111] This embodiment provides a linear attention mechanism method for visual tasks, which can be used in the aforementioned electronic devices or terminal devices. Figure 4 This is a flowchart of a linear attention mechanism method for vision tasks according to an embodiment of the present invention, as shown below. Figure 4 As shown, the process includes the following steps:

[0112] Step S401 involves performing norm decomposition and direction decomposition on the input query vector and key vector, respectively, to obtain the magnitude and unit direction vector corresponding to the query vector and key vector. For details, please refer to [link to relevant documentation]. Figure 3 Step S301 of the illustrated embodiment will not be described again here.

[0113] Step S402: Based on the magnitude of the query vector, a magnitude-aware kernel function is used to dynamically modulate the query unit direction vector, and a fixed magnitude modulation is performed on the key unit direction vector based on the magnitude of the key vector. For details, please refer to [link to details]. Figure 3 Step S302 of the illustrated embodiment will not be described again here.

[0114] Step S403: Map the query unit direction vector and the key unit direction vector to trigonometric function features respectively to obtain the query direction feature vector and the key direction feature vector.

[0115] Specifically, this embodiment is based on the non-negativity constraint of cosine suppression. The cosine suppression mechanism is the core of this invention in resolving the contradiction between the non-negativity constraint and the preservation of semantic interaction information in linear attention. Its design goal is to fully preserve the directional interaction features between the query and key vectors while ensuring the kernel function output is non-negative, especially avoiding the loss of negative semantic relationships in traditional methods. This mechanism achieves fine-grained control over vector directional interactions through the collaborative design of trigonometric function mapping and cosine difference similarity calculation.

[0116] In its implementation, the cosine suppression mechanism consists of two key steps. The first step is the trigonometric function mapping of the direction vector. For the unit direction vector d(x) (including the query) obtained through norm-direction decomposition, a nonnegative mapping function is used. This is transformed into trigonometric function features. This mapping utilizes the boundedness of trigonometric functions (the output range is always between [-1, 1]) to control the numerical fluctuations of the original direction vector within a stable range. Unlike activation functions such as ReLU that directly filter negative values, this mapping does not discard any original direction information; it simply transforms it into a feature form more suitable for non-negativity processing. Step S403 is the specific implementation of the first step, and step S403 includes:

[0117] Step S4031: Obtain each component of the query unit direction vector, calculate the cosine and sine values ​​respectively, and arrange all the calculated cosine values ​​in order to form the query cosine branch vector, and arrange all the calculated sine values ​​in order to form the query sine branch vector; concatenate the query cosine branch vector and the query sine branch vector to obtain the query direction feature vector.

[0118] Through the above nonnegative mapping function Each component of the query unit direction vector is transformed into a trigonometric function feature. Specifically, each component of the query unit direction vector is obtained, and its cosine and sine values ​​are calculated separately. All cosine values ​​are arranged in their original order to form a query cosine branch vector, and all sine values ​​are arranged in their original order to form a query sine branch vector. These two branch vectors are then concatenated to obtain a query direction feature vector that contains both cosine and sine information. This process fully preserves all component information of the direction vector through trigonometric function mapping, providing an accurate direction feature foundation for subsequent fusion with amplitude modulation results.

[0119] Step S4032: Obtain each component of the bond unit direction vector, calculate the cosine and sine values ​​respectively, and arrange all the calculated cosine values ​​in order to form the bond cosine branch vector, and arrange all the calculated sine values ​​in order to form the bond sine branch vector; concatenate the bond cosine branch vector and the bond sine branch vector to obtain the bond direction feature vector.

[0120] This step also involves the aforementioned nonnegative mapping function. Each component of the bond unit direction vector is transformed into a trigonometric function feature. Specifically, each component of the bond unit direction vector is obtained, and its cosine and sine values ​​are calculated separately. All cosine values ​​are arranged in their original order to form a bond cosine branch vector, and all sine values ​​are arranged in their original order to form a bond sine branch vector. These two branch vectors are then concatenated to obtain a bond direction feature vector that contains both cosine and sine information. This process fully preserves all component information of the direction vector through trigonometric function mapping, providing an accurate direction feature foundation for subsequent fusion with the bond magnitude scalar.

[0121] Step S404: The modulated query amplitude information is fused with the query direction vector feature to obtain the query kernel function feature vector; the modulated key amplitude information is fused with the key direction feature to obtain the key kernel function feature vector.

[0122] Specifically, the second step of the cosine suppression mechanism is a similarity calculation based on the cosine difference formula, through... Implement interactive modeling of query and key direction components. Based on the cosine difference formula of trigonometric functions. This calculation is essentially an inner product operation between the mapped query and key features, but its result has a clear directional interaction semantic: when the directional components of the query and key... and When they are in the same direction (angle difference close to 0), the cosine value is close to 1, corresponding to a strong positive interaction weight; when they are in opposite directions (angle difference close to 1), the cosine value is close to 1. The cosine value is close to -1, but it is transformed into a weak interaction weight through subsequent absolute value processing (incorporating the non-negativity requirement of the kernel function); when the direction is perpendicular (the angle difference is close to 1), the cosine value is close to 1. The cosine value is close to 0, corresponding to negligible interaction effects. This mechanism achieves differentiated processing of "enhancement in the same direction and suppression in the opposite direction," which satisfies the requirement of non-negativity of the kernel function for linear attention while preserving the polarity difference information of directional interactions.

[0123] The similarity calculation based on the cosine difference formula is the theoretical basis for the directional interaction of this invention. It mathematically proves that the inner product of the query and key trigonometric function features is equivalent to calculating the cosine value of the angle difference, thereby achieving precise control over "in-direction enhancement and reverse suppression". Step S404 is the specific implementation step of this theory. By multiplying the query amplitude modulation vector element-wise with the query's cosine and sine branches and concatenating them, the query's intensity and direction information are fused into the query kernel function feature vector. Similarly, the key kernel function feature vector is obtained, so that the subsequent inner product calculation of these two feature vectors can actually perform the similarity measurement described by the cosine difference formula, thus completely reproducing the interaction effect of the theoretical design at the engineering implementation level.

[0124] Step S404 above includes:

[0125] Step S4041: Multiply the query amplitude modulation vector and the query cosine branch vector element by element to obtain the modulated query cosine branch; multiply the query amplitude modulation vector and the query sine branch vector element by element to obtain the modulated query sine branch; concatenate the modulated query cosine branch and the modulated query sine branch to obtain the query kernel function feature vector.

[0126] For example, suppose:

[0127] The query unit direction vector is two-dimensional: d ( q = [0.6, 0.8];

[0128] The query amplitude modulation vector is (dynamically calculated based on the query modulus): a q =[1.2,1.5].

[0129] Step 1: Calculate the query direction feature vector:

[0130] Cosine branch: cos( d ( q ))=[cos(0.6),cos(0.8)]≈[0.8253,0.6967];

[0131] Sine branch: sin( d ( q ))=[sin(0.6),sin(0.8)]≈[0.5646,0.7174];

[0132] Step 2: Multiply the amplitude modulation vector element-wise with the cosine branch:

[0133] Modulated cosine branch: a q ⊙cos( d ( q =[1.2×0.8253,1.5×0.6967]=[0.9904,1.0451];

[0134] Step 3: Multiply the amplitude modulation vector element-wise with the sine branch:

[0135] Modulated sine branch: a q ⊙sin( d ( q =[1.2×0.5646,1.5×0.7174]=[0.6775,1.0761];

[0136] Step 4: Concatenate to obtain the query kernel function feature vector:

[0137] The query kernel function eigenvector = [modulated cosine branch, modulated sine branch] = [0.9904, 1.0451, 0.6775, 1.0761].

[0138] Step S4042: Multiply the bond amplitude scalar with the bond cosine branch vector element by element to obtain the modulated bond cosine branch; multiply the bond amplitude scalar with the bond sine branch vector element by element to obtain the modulated bond sine branch; concatenate the modulated bond cosine branch and the modulated bond sine branch to obtain the bond kernel function feature vector.

[0139] For example, assume the bond unit direction vector is two-dimensional: d ( k = [0.3, 0.4], the magnitude of the key vector. =2.5, with a preset fixed power λ=2, then the key magnitude scalar m k =(2.5)2=6.25.

[0140] Calculate the bond direction eigenvector:

[0141] Cosine branch: cos( d ( k ))=[cos(0.3),cos(0.4)]≈[0.9553,0.9211];

[0142] Sine branch: sin( d ( k ))=[sin(0.3),sin(0.4)]≈[0.2955,0.3894];

[0143] Multiply the key magnitude scalar by the cosine branch element by element:

[0144] Modulated key cosine branch: 6.25 × [0.9553, 0.9211] = [5.9706, 5.7569];

[0145] Multiply the key magnitude scalar by the sine branch element by element:

[0146] Modulated key sine branch: 6.25 × [0.2955, 0.3894] = [1.8469, 2.4338];

[0147] The key kernel function eigenvectors are obtained by concatenation:

[0148] Key kernel function features = [5.9706, 5.7569, 1.8469, 2.4338].

[0149] Step S405: Based on the query kernel function feature vector and the key kernel function feature vector, calculate the linear attention result.

[0150] Specifically, step S405 includes:

[0151] Step S4051: Calculate the matrix multiplication of the query kernel function feature vector and the transpose of all key kernel function feature vectors to obtain the original attention score matrix.

[0152] Specifically, the feature vector matrix of the current input query kernel function is multiplied by the transpose of the feature vector matrices of all key kernel functions to obtain the original attention score matrix. Each element in this matrix corresponds to the original interaction score between a query position and a key position, reflecting the degree of similarity between the query and the key before normalization, and providing a basic similarity measure for the subsequent calculation of attention weights.

[0153] Step S4052: Calculate the matrix multiplication of the query kernel function eigenvector and the transpose of all key kernel function eigenvectors to obtain the normalized denominator vector.

[0154] Specifically, the query kernel function feature vector matrix is ​​multiplied by the transpose of all key kernel function feature vector matrices to obtain a normalized denominator vector. Each element in this vector corresponds to the sum of the original scores of a query position and all key positions, which serves as the denominator for subsequent position-by-position normalization. This ensures that the sum of the attention weights corresponding to each query is 1, satisfying the normative requirements of the probability distribution.

[0155] Step S4053: Divide the original attention score matrix and the normalized denominator vector position by position to obtain the normalized attention weight matrix.

[0156] Specifically, each element in the original attention score matrix is ​​divided positionally by the element corresponding to the query position in the normalized denominator vector to obtain the normalized attention weight matrix. Each element in this matrix represents the final attention weight of a query position to a key position. The weight value is between 0 and 1, and the sum of all weights corresponding to each query is 1, thus achieving the normalization of the original scores.

[0157] Step S4054: Perform matrix multiplication between the normalized attention weight matrix and the pre-calculated value vector matrix to obtain the weighted summation output feature matrix, and output the output feature matrix as the result of linear attention calculation.

[0158] Specifically, the normalized attention weight matrix is ​​multiplied by the pre-calculated value vector matrix to obtain a weighted summation output feature matrix. Each element in this matrix is ​​the result of weighted aggregation of value vectors according to attention weights, representing the fusion information extracted from the value vectors of all key positions for each query position. Finally, this output feature matrix is ​​output as the result of linear attention calculation, completing the entire linear attention calculation process.

[0159] The output of linear attention is:

[0160] ;

[0161] in, This is the output vector for the t-th query position, i.e., the output result of linear attention; For query vector The kernel function vector; Key vector The transpose of the kernel function vector; To query the cosine branch of the kernel function's eigenvectors; To query the sine branch of the kernel function's eigenvectors; The cosine branch of the eigenvectors of the key kernel function; This represents the sinusoidal branch of the eigenvectors of the bond kernel function; For element-wise multiplication; This is the gated vector.

[0162] The linear attention mechanism method for vision tasks provided in this embodiment offers an attention mechanism that can restore the dynamic entropy reduction characteristics of norm awareness and the full vector interaction capability while maintaining linear complexity.

[0163] As one or more specific application embodiments of the present invention, combined with Figures 5 to 7 The linear attention mechanism method for vision tasks provided by this invention will be further described in detail, such as... Figure 7 As shown, the specific process is as follows:

[0164] Figure 7 This demonstrates the linear attention computation process from the input sequence X to the final output:

[0165] 1. Input sequence: X serves as the input to the entire attention mechanism.

[0166] 2. Linear Projection: Input X undergoes three different linear transformations to generate query vector Q, key vector K, and value vector V, respectively.

[0167] 3. Key-value interaction computation:

[0168] The key vector K is mapped by a kernel function to obtain the key kernel function eigenvector. k(K).

[0169] Will Perform matrix multiplication (or specific combination operations) between k(K) and the value vector V to obtain an intermediate result. k(K)V.

[0170] 4. Attention Output Calculation: Combine the kernel function features of the query vector Q with the above intermediate results (usually through matrix multiplication and normalization) to finally generate the attention output.

[0171] For example:

[0172] A schematic diagram of the dynamic action of the modulus-sensing kernel function is shown below. Figure 5 As shown in the attached diagram, the diagram includes two parallel processing branches, corresponding to the processing flow of the query vector Q and the key vector K, respectively, which are finally combined with the value vector V to calculate the output.

[0173] Query branch:

[0174] 1. Input: Query vector Q.

[0175] 2. Norm and Directional Decomposition: Calculating the Modulus of Q And obtain the unit direction vector. .

[0176] 3. Dynamic amplitude modulation: based on the modulus... Through formula Calculate dynamic exponentiation .

[0177] For unit direction vector Perform exponentiation to obtain (The absolute value is taken to ensure non-negativity).

[0178] 4. Cosine suppression module: Reduces the unit direction vector... Input the cosine suppression module, perform trigonometric function mapping, and obtain features containing cosine and sine information.

[0179] 5. Fusion: The amplitude modulation result is fused with the directional features output by the cosine suppression module to obtain the final query kernel function feature vector. .

[0180] Key branch:

[0181] 1. Input: Key vector K.

[0182] 2. Norm and Directional Decomposition: Calculating the Modulus of K And obtain the unit direction vector. .

[0183] 3. Dynamic amplitude modulation: This refers to the modulation of the modulus. Perform fixed-power operations (power is a preset hyperparameter). ),get (Take the absolute value).

[0184] 4. Cosine suppression module: Reduces the unit direction vector... Input the cosine suppression module, perform trigonometric function mapping, and obtain features containing cosine and sine information.

[0185] 5. Fusion: The fixed amplitude modulation result is fused with the directional features output by the cosine suppression module to obtain the final bond kernel function feature vector. .

[0186] Output calculation: query kernel feature vector q (Q) and the eigenvector of the bond kernel function k (K) performs interactive computation (inner product) and combines it with the value vector. V This yields the final linear attention result. O .

[0187] A schematic diagram of the nonnegativity preservation process of cosine suppression is shown below. Figure 6 As shown, direction preprocessing: by scaling with tanh and π / 4, the unit direction vector is mapped to a finite range of angles suitable for trigonometric function calculations.

[0188] Trigonometric function mapping: Calculate the cosine and sine values ​​of the preprocessed angles to generate cosine and sine branches.

[0189] Feature combination: The cosine and sine branches together form the directional feature vector, which provides accurate input for subsequent fusion with amplitude modulation results and directional interactive calculation.

[0190] Figure 6 for Figure 5 The internal expansion details of the cosine suppression module demonstrate the specific implementation path of how to generate trigonometric function eigenvectors from unit direction vectors, including:

[0191] The system has two parallel branches, one above and one below, which correspond to the query and key processing flows, respectively. Finally, a directional feature vector is generated for subsequent interactions through trigonometric function mapping.

[0192] Query branch:

[0193] 1. Input: Query vector Q.

[0194] 2. Norm and Directional Decomposition: Calculating the Modulus of Q And obtain the unit direction vector. .

[0195] 3. Direction preprocessing: For unit direction vectors Perform the tanh() transformation and multiply by π / 4 to obtain the adjusted direction representation d(q).

[0196] 4. Trigonometric function mapping:

[0197] Input the adjusted direction d(q) into the cosine function cos() and the sine function sin() respectively.

[0198] Output the cosine component cos(d(q)) and the sine component sin(d(q)).

[0199] 5. Feature Combination: Combine the cosine and sine components to obtain the query direction feature vector. ( q (The splicing operation is not explicitly shown in the diagram, but [cos( x ),sin( x [This implies a combination relationship.]

[0200] Key branch:

[0201] 1. Input: Key vector K.

[0202] 2. Norm and Directional Decomposition: Calculating the Modulus of K And obtain the unit direction vector d(K).

[0203] 3. Direction preprocessing: Perform tanh() transformation on the unit direction vector d(K) and multiply by π / 4 to obtain the adjusted direction representation d(k).

[0204] 4. Trigonometric function mapping:

[0205] Input the adjusted direction d(k) into the cosine function cos() and the sine function sin() respectively.

[0206] Output cosine component cos(d( k )) and the sinusoidal component sin(d( k )).

[0207] 5. Feature Combination: Combining the cosine and sine components yields the bond direction feature vector. (k) (Similarly, [cos( x ),sin( x [This implies a combination relationship.]

[0208] Subsequent processing: generated (q) and (k) will be fused with the corresponding amplitude modulation results to form the final kernel function feature vector (this part is completed outside the figure).

[0209] This embodiment validates the effectiveness of the proposed method on visual and language tasks. For visual tasks, this embodiment conducts image classification experiments on the ImageNet-1K dataset, object detection and instance segmentation experiments on the COCO dataset, and semantic segmentation experiments on the ADE20K dataset, comparing the performance with current high-performance models. For language tasks, this embodiment pre-trains a language model from scratch and evaluates the pre-trained model on a commonsense reasoning task.

[0210] In the image classification experiments, this embodiment trains the model from scratch on the ImageNet-1K dataset and compares the Top-1 accuracy (Acc). The baseline model is divided into four classes based on its parameter size (Param) and floating-point operations (FLOPs), and performance is then compared within each group. Compared to the baseline model, the proposed method consistently demonstrates higher accuracy across models of various sizes.

[0211] This embodiment also provides a linear attention mechanism device for visual tasks, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0212] This embodiment provides a linear attention mechanism device for visual tasks, such as... Figure 8 As shown, it includes:

[0213] The norm and direction decomposition module 801 is used to perform norm decomposition and direction decomposition on the input query vector and key vector respectively, to obtain the magnitude and unit direction vector corresponding to the query vector and key vector.

[0214] The module 802 is used to dynamically modulate the query unit direction vector based on the magnitude of the query vector using the module length sensing kernel function, and to perform fixed magnitude modulation on the key unit direction vector based on the magnitude of the key vector.

[0215] The direction mapping module 803 is used to map the query unit direction vector and the key unit direction vector to trigonometric function features, respectively, to obtain the query direction feature vector and the key direction feature vector.

[0216] The direction interaction calculation module 804 is used to fuse the modulated query amplitude information with the query direction vector feature to obtain the query kernel function feature vector; and to fuse the modulated key amplitude information with the key direction feature to obtain the key kernel function feature vector.

[0217] The linear attention output module 805 is used to calculate the linear attention result based on the query kernel function feature vector and the key kernel function feature vector.

[0218] In some alternative implementations, the norm and direction decomposition module 801 includes:

[0219] The query vector norm and direction decomposition unit are used to calculate the magnitude of the query vector and, based on the query vector and its corresponding magnitude, to calculate the query unit direction vector.

[0220] The bond vector norm and direction decomposition unit are used to calculate the magnitude of the bond vector and, based on the bond vector and its corresponding magnitude, to calculate the bond unit direction vector.

[0221] In some alternative implementations, the modulus-sensing kernel function module 802 includes:

[0222] The modulus-sensing kernel function calculation unit is used to calculate the dynamic modulation power based on the modulus of the query vector using the modulus-sensing kernel function;

[0223] The dynamic exponentiation unit is used to perform exponentiation on each component of the query unit direction vector according to the dynamic modulation exponentiation, and combine the components after exponentiation to obtain the query amplitude modulation vector.

[0224] The fixed power operation unit is used to take the magnitude of the key vector as the input of fixed amplitude modulation, perform power operation on the magnitude of the key vector using a preset fixed power, and use the result of the power operation as the key amplitude scalar.

[0225] In some alternative implementations, the orientation mapping module 803 includes:

[0226] The query vector mapping unit is used to obtain each component of the query unit direction vector, calculate the cosine and sine values ​​respectively, and arrange all the calculated cosine values ​​in order to form the query cosine branch vector, and arrange all the calculated sine values ​​in order to form the query sine branch vector; the query cosine branch vector and the query sine branch vector are concatenated to obtain the query direction feature vector.

[0227] The bond vector mapping unit is used to obtain each component of the bond unit direction vector, calculate the cosine and sine values ​​respectively, and arrange all the calculated cosine values ​​in order to form the bond cosine branch vector, and arrange all the calculated sine values ​​in order to form the bond sine branch vector; the bond cosine branch vector and the bond sine branch vector are concatenated to obtain the bond direction feature vector.

[0228] In some alternative implementations, the direction interaction calculation module 804 includes:

[0229] The query kernel function feature vector calculation unit is used to multiply the query amplitude modulation vector and the query cosine branch vector element-wise to obtain the modulated query cosine branch; multiply the query amplitude modulation vector and the query sine branch vector element-wise to obtain the modulated query sine branch; and concatenate the modulated query cosine branch and the modulated query sine branch to obtain the query kernel function feature vector.

[0230] The key kernel function eigenvector calculation unit is used to multiply the key amplitude scalar with the key cosine branch vector element by element to obtain the modulated key cosine branch; multiply the key amplitude scalar with the key sine branch vector element by element to obtain the modulated key sine branch; and concatenate the modulated key cosine branch with the modulated key sine branch to obtain the key kernel function eigenvector.

[0231] In some alternative implementations, the linear attention output module 805 includes:

[0232] The first computational unit is used to calculate the matrix multiplication of the query kernel function eigenvector and the transpose of all key kernel function eigenvectors to obtain the original attention score matrix.

[0233] The second calculation unit is used to calculate the matrix multiplication of the query kernel function eigenvector and the transpose of the eigenvectors of all key kernel functions to obtain the normalized denominator vector.

[0234] The third calculation unit is used to divide the original attention score matrix position by position with the normalized denominator vector to obtain the normalized attention weight matrix.

[0235] The fourth calculation unit is used to perform matrix multiplication between the normalized attention weight matrix and the pre-calculated value vector matrix to obtain the weighted summation of the output feature matrix, and output the output feature matrix as the result of linear attention calculation.

[0236] The linear attention mechanism apparatus for vision tasks provided in this embodiment of the invention can execute the linear attention mechanism method for vision tasks provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the various modules and units described above are the same as in the corresponding embodiments described above, and will not be repeated here.

[0237] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0238] The following is a detailed reference. Figure 9This diagram illustrates a structural schematic suitable for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 901, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 902 or a program loaded from memory 908 into random access memory (RAM) 903. The RAM 903 also stores various programs and data required for the operation of the electronic device. The processor 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0239] Typically, the following devices can be connected to I / O interface 905: input devices 906 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 907 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 908 including, for example, magnetic tapes, hard disks, etc.; and communication devices 909. Communication device 909 allows electronic devices to exchange data via wireless or wired communication with other devices. Although Figure 9 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0240] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 909, or installed from a memory 908, or installed from a ROM 902. When the computer program is executed by the processor 901, it performs the functions defined in the linear attention mechanism method for vision tasks according to embodiments of the present invention.

[0241] Figure 9 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0242] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the linear attention mechanism method for vision tasks shown in the above embodiments is implemented.

[0243] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0244] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A linear attention mechanism method for visual tasks, characterized in that, The method includes: Perform norm decomposition and direction decomposition on the input query vector and key vector respectively to obtain the magnitude and unit direction vector corresponding to the query vector and key vector; Based on the magnitude of the query vector, a magnitude-aware kernel function is used to dynamically modulate the query unit direction vector, and a fixed magnitude modulation is performed on the key unit direction vector based on the magnitude of the key vector. The query unit direction vector and the key unit direction vector are mapped to trigonometric function features respectively to obtain query direction feature vector and key direction feature vector, including: Obtain each component of the query unit direction vector, calculate the cosine and sine values ​​respectively, and arrange all the calculated cosine values ​​in order to form the query cosine branch vector, and arrange all the calculated sine values ​​in order to form the query sine branch vector; The query cosine branch vector and the query sine branch vector are concatenated to obtain the query direction feature vector; Obtain each component of the bond unit direction vector, calculate the cosine and sine values ​​respectively, and arrange all the calculated cosine values ​​in order to form the bond cosine branch vector, and arrange all the calculated sine values ​​in order to form the bond sine branch vector; The bond cosine branch vector and the bond sine branch vector are concatenated to obtain the bond direction feature vector; The modulated query amplitude information is fused with the query direction feature vector to obtain the query kernel function feature vector, including: Multiply the query amplitude modulation vector and the query cosine branch vector element by element to obtain the modulated query cosine branch; Multiply the query amplitude modulation vector and the query sine branch vector element by element to obtain the modulated query sine branch; The modulated query cosine branch and the modulated query sine branch are concatenated to obtain the query kernel function feature vector; The modulated bond amplitude information is fused with the bond direction feature vector to obtain the bond kernel function feature vector; Based on the query kernel function feature vector and the key kernel function feature vector, the linear attention result is calculated, including: The original attention matrix is ​​obtained by multiplying the transpose of the eigenvectors of all key kernel functions with the matrix multiplication of the value vectors, and then multiplying the matrix with the eigenvectors of the query kernel function. Calculate the matrix multiplication of the query kernel function eigenvector and the transpose of the eigenvectors of all key kernel functions to obtain the normalized denominator vector; The original attention matrix is ​​divided position by position by the normalized denominator vector to obtain the normalized output feature matrix, which is then output as the result of the linear attention calculation.

2. The method according to claim 1, characterized in that, The process of performing norm decomposition and direction decomposition on the input query vector and key vector respectively to obtain the magnitude and unit direction vector corresponding to the query vector and key vector includes: Calculate the magnitude of the query vector, and based on the query vector and its corresponding magnitude, calculate the query unit direction vector; Calculate the magnitude of the bond vector, and based on the bond vector and its corresponding magnitude, calculate the bond unit direction vector.

3. The method according to claim 1, characterized in that, Based on the magnitude of the query vector, a magnitude-aware kernel function is used to dynamically modulate the magnitude of the query unit direction vector, including: Based on the magnitude of the query vector, the dynamic modulation power is calculated using the magnitude-aware kernel function; According to the dynamic modulation power, each component of the query unit direction vector is subjected to a power operation, and the components after the power operation are combined to obtain the query amplitude modulation vector.

4. The method according to claim 3, characterized in that, Based on the magnitude of the bond vector, the unit direction vector of the bond is modulated with a fixed amplitude, including: The magnitude of the key vector is used as the input for fixed amplitude modulation. A preset fixed power is used to exponentiate the magnitude of the key vector, and the result of the power operation is used as the key amplitude scalar.

5. The method according to claim 1, characterized in that, The modulated bond amplitude information is fused with the bond direction feature vector to obtain the bond kernel function feature vector, including: The modulated key cosine branch is obtained by multiplying the key amplitude scalar with the key cosine branch vector element by element. The modulated key sinusoidal branch is obtained by multiplying the key amplitude scalar with the key sinusoidal branch vector element by element. The modulated key cosine branch and the modulated key sine branch are concatenated to obtain the key kernel function feature vector.

6. A linear attention mechanism device for visual tasks, characterized in that, The apparatus for the linear attention mechanism method for vision-oriented tasks according to any one of claims 1 to 5 comprises: The norm and direction decomposition module is used to perform norm decomposition and direction decomposition on the input query vector and key vector respectively, to obtain the magnitude and unit direction vector of the query vector and key vector; The module with modulus-aware kernel function is used to dynamically modulate the query unit direction vector based on the modulus of the query vector, and to perform fixed amplitude modulation on the key unit direction vector based on the modulus of the key vector. The direction mapping module is used to map the query unit direction vector and the key unit direction vector to trigonometric function features respectively, so as to obtain the query direction feature vector and the key direction feature vector; The direction interaction calculation module is used to fuse the modulated query amplitude information with the query direction feature vector to obtain the query kernel function feature vector; and to fuse the modulated key amplitude information with the key direction feature vector to obtain the key kernel function feature vector. The linear attention output module is used to calculate the linear attention result based on the query kernel function feature vector and the key kernel function feature vector.

7. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory storing computer instructions, and the processor executing the computer instructions to perform the linear attention mechanism method for vision tasks as described in any one of claims 1 to 5.

Images