Attention feature extraction method and device and computer readable storage medium

Abstract

The embodiment of the invention provides an attention feature extraction method and device and a computer readable storage medium. The method comprises the following steps: calculating a fixed-point attention weight vector of a to-be-queried content to a reference content according to the to-be-queried content and the reference content; the normalization result of the fixed-point attention weight vector is expressed as that the floating-point middle attention weight vector is divided by the element sum of the floating-point middle attention weight vector; performing quantization processing on the floating point intermediate attention weight vector, the floating point feature vector of the reference content and the element sum of the floating point intermediate attention weight vector; and performing point multiplication on the fixed-point intermediate attention weight vector and the fixed-point feature vector of the reference content, then dividing the product by the fixed-point value of the element sum of the floating-point intermediate attention weight vector, and taking the obtained fixed-point vector of the quotient vector as the fixed-point attention feature vector of the to-be-queried content. According to the embodiment of the invention, the quantization loss in the attention feature extraction process is reduced while the attention feature extraction complexity is not increased.

Description

Technical Field

[0001] The present invention relates to the field of feature extraction technology, and in particular to an attention feature extraction method, device and computer-readable storage medium. Background Art

[0002] At present, deep learning models have been widely used in various fields, such as computer vision (CV, Computer Vision), natural language processing (NLP, Natural Language Processing), speech processing (such as automatic speech recognition (ASR, Automatic Speech Recognition), text to speech (TTS, Text-To-Speech)) and other fields.

[0003] The Transformer model is a deep learning model based on the self-attention mechanism. An important feature of the Transformer model is its attention mechanism. There is an Attention module in each block of the Transformer model. The Attention module enables the model to pay attention to the importance of different parts of the input sequence, thereby effectively improving the performance of the model.

[0004] Model quantization is a commonly used neural network model acceleration algorithm. By converting the model's float type parameters and input and output feature values ​​into fixed-bit values, such as 8 bits and 4 bits, the computational cost, data bandwidth, and storage space of the neural network model can be reduced, allowing the neural network model to be applied faster and better on hardware with limited memory and computing power. The fixed-point model includes the fixed-point model weights and feature values. The lower the number of fixed-point bits, the more obvious the acceleration on the hardware, and the performance will be reduced accordingly. Transformer models usually have a large number of parameters and computational complexity. Therefore, in the application of Transformer models, quantization is often required to reduce the resource overhead of model deployment and reasoning. The Attention module usually introduces a large performance loss during the quantization process, which affects the performance of Transformer model deployment and reasoning.

[0005] In the existing scheme, when the Transformer model is used to infer the target floating-point data, the quantized attention weights are normalized again to reduce the error caused by the attention weights during the inference process. Furthermore, the normalization factor is used to adjust the output fixed-point data features to achieve feature correction and obtain the target fixed-point data features, so that the inference results based on the target fixed-point data features are more accurate. This scheme aims to dynamically correct the quantization deviation of the Attention module based on the characteristic that the sum of the normalized output values ​​is 1 by introducing a normalization factor, thereby alleviating the quantization error of the Attention module. The disadvantage of this scheme is that the deviation correction occurs only after the Attention quantization accuracy has been lost, and the lost information is difficult to completely compensate, so there is still a drop in performance. In addition, the normalization method introduces new operations on the original module, resulting in an increase in computational complexity. Summary of the invention

[0006] The embodiments of the present invention propose an attention feature extraction method and device to reduce the quantization loss in the attention feature extraction process without increasing the complexity of the attention feature extraction; the embodiments of the present invention also propose a non-transient computer-readable storage medium and an electronic device to reduce the quantization loss in the attention feature extraction process without increasing the complexity of the attention feature extraction.

[0007] The technical solution of the embodiment of the present invention is achieved as follows: A method for extracting attention features, the method comprising: Obtaining the content to be queried, the reference content, and the floating-point feature vector of the reference content; According to the content to be queried and the reference content, a fixed-point attention weight vector of the content to be queried to the reference content is calculated; The normalized result of the fixed-point attention weight vector is expressed as: the floating-point intermediate attention weight vector divided by the element-wise sum of the floating-point intermediate attention weight vector; Quantize the floating-point intermediate attention weight vector, the floating-point feature vector of the reference content, and the element sum of the floating-point intermediate attention weight vector to obtain a fixed-point value of the element sum of the fixed-point intermediate attention weight vector, the fixed-point feature vector of the reference content, and the floating-point intermediate attention weight vector; The fixed-point intermediate attention weight vector is multiplied by the fixed-point feature vector of the reference content, and then divided by the fixed-point value of the sum of the elements of the floating-point intermediate attention weight vector, and the fixed-point vector of the obtained quotient vector is used as the fixed-point attention feature vector of the content to be queried.

[0008] Each floating point intermediate attention weight in the floating point intermediate attention weight vector is obtained as follows: Find the maximum value in the fixed-point attention weight vector; Calculate the difference between the fixed-point attention weight corresponding to the floating-point intermediate attention weight and the maximum value; Calculate e The power with the difference as the base is the floating-point intermediate attention weight.

[0009] The step of calculating the fixed-point attention weight vector of the content to be queried for the reference content according to the content to be queried and the reference content includes: The fixed-point vector of the content vector to be queried and the fixed-point vector of the transposed vector of the reference content vector are point-multiplied and then divided by the square root of the dimension of the reference content vector, and the fixed-point vector of the obtained quotient vector is used as the fixed-point attention weight vector.

[0010] After the obtained fixed-point vector of the quotient vector is used as the fixed-point attention feature vector of the content to be queried, the method further includes: According to the fixed-point attention feature vector of the content to be queried, the query task for the content to be queried is performed.

[0011] The to-be-queried content and the reference content are images; The query task is to query whether a certain target exists in the image to be queried, or to query the position of the target in the image to be queried.

[0012] The content to be queried and the reference content are both texts; The query task is to translate the text to be queried into another language text, or to extract target summary information from the text to be queried.

[0013] The to-be-queried content and the reference content are both voice; The query task is to identify target information in the voice to be queried, or to query whether the voice to be queried is the voice of a target person.

[0014] An attention feature extraction device, the device comprising: A content acquisition module, which acquires the content to be queried, the reference content and the floating-point feature vector of the reference content; The feature extraction module calculates the fixed-point attention weight vector of the content to be queried for the reference content based on the content to be queried and the reference content; expresses the normalized result of the fixed-point attention weight vector as: the floating-point intermediate attention weight vector divided by the element sum of the floating-point intermediate attention weight vector; quantizes the floating-point intermediate attention weight vector, the floating-point feature vector of the reference content, and the element sum of the floating-point intermediate attention weight vector, respectively, to obtain the fixed-point value of the fixed-point intermediate attention weight vector, the fixed-point feature vector of the reference content, and the element sum of the floating-point intermediate attention weight vector; performs dot multiplication on the fixed-point intermediate attention weight vector and the fixed-point feature vector of the reference content, and then divides it by the fixed-point value of the element sum of the floating-point intermediate attention weight vector, and uses the fixed-point vector of the quotient vector as the fixed-point attention feature vector of the content to be queried.

[0015] Each floating-point intermediate attention weight in the floating-point intermediate attention weight vector in the feature extraction module is obtained as follows: Find the maximum value in the fixed-point attention weight vector; Calculate the difference between the fixed-point attention weight corresponding to the floating-point intermediate attention weight and the maximum value; Calculate e The power with the difference as the base is the floating-point intermediate attention weight.

[0016] After the feature extraction module uses the obtained fixed-point vector of the quotient vector as the fixed-point attention feature vector of the content to be queried, the feature extraction module further includes: According to the fixed-point attention feature vector of the content to be queried, the query task for the content to be queried is performed.

[0017] The content to be queried and the reference content are images, and the feature extraction module performs a query task on the content to be queried, including: querying whether a certain target exists in the image to be queried, or querying the position of the target in the image to be queried; Alternatively, the content to be queried and the reference content are both texts, and the feature extraction module performs a query task on the content to be queried, including: translating the text to be queried into another type of language text, or extracting target summary information from the text to be queried; Alternatively, the content to be queried and the reference content are both voices, and the feature extraction module performs a query task on the content to be queried, including: identifying target information in the voice to be queried, or querying whether the voice to be queried is the voice of a target person.

[0018] A non-transitory computer-readable storage medium storing instructions which, when executed by a processor, cause the processor to perform any of the attention feature extraction methods described above.

[0019] In the above embodiment, the normalized result of the fixed-point attention weight vector is expressed as: the floating-point intermediate attention weight vector divided by the element sum of the floating-point intermediate attention weight vector, and then the floating-point intermediate attention weight vector, the floating-point feature vector of the reference content and the element sum of the floating-point intermediate attention weight vector are quantized respectively, and then the fixed-point intermediate attention weight vector and the fixed-point feature vector of the reference content are dot-multiplied and then divided by the fixed-point value of the element sum of the floating-point intermediate attention weight vector, and the fixed-point vector of the obtained quotient vector is used as the fixed-point attention feature vector of the content to be queried, thereby reducing the quantization loss in the attention feature extraction process without increasing the complexity of the attention feature extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0021] Figure 1 For a practical application scenario, Softmax Schematic diagram of the distribution of element values ​​of (); Figure 2 A flow chart of an attention feature extraction method provided by an embodiment of the present invention; Figure 3 A schematic diagram of the structure of an attention feature extraction device provided in one embodiment of the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] In order to solve the technical problem proposed by the present invention, the inventor first analyzed the calculation process of the Attention module in the existing Transformer model: In the Transformer model, the calculation of the Attention module can be expressed as: (1) in, Q , K , V Represent three important input vectors in the Attention module: Query (Query) Vector 、 Key (Key) Vector and Value a vector of (values), d express Key The dimension of the vector, Softmax () is the normalization function, Attn ( Q , K , V ) represents the attention feature vector. The entire calculation process of the Attention module can be described as: 1) Use Query Vector Sum Key Vector calculation attention weight vector P , calculated as: Q Vector and K The transpose of the vector is used for dot multiplication. This is to scale the point product result to prevent the gradient from disappearing or exploding. To simplify the subsequent description, Recorded as P ; 2) Attention weight vector P use Softmax () function to normalize and obtain the final attention score vector; 3) Use the attention score vector with Value The vectors are weighted summed to generate the final attention feature vector and output it.

[0025] In actual deployment, in order to reduce the resource overhead of model deployment and inference, the model needs to be quantized. The quantized Attention module is expressed as:

[0026] in, Quant () represents the quantization function, that is, the quantized Attention module needs to Q , K , V The vector and the calculation results of each step are quantized to ensure the quantized inference speed of the Attention module on the actual hardware. Quant There are two ways to calculate (): Method 1: ; Method 2: .

[0027] in, data represents the vector to be quantized; bw Indicates the quantization bit width, usually 8, indicating that 8-bit quantization is used; param Represents quantized hyperparameters; clip () indicates a truncation function, such as: clip ( a , min , max ) means a Restricted to [ min , max ] within the scope; round () indicates the rounding function.

[0028] Among them, the quantized hyperparameter param This can be calculated in one of two ways: Method 1: ; Method 2: ; in, max for data The maximum value in ceil () is the rounding up function.

[0029] Generally speaking, , , , The quantization loss at 8 bits is relatively small. After observation and analysis, it is found that the quantization loss of the Attention module mainly comes from Softmax () quantization process, that is .

[0030] Softmax () has this characteristic: the sum of all elements is 1, so Softmax The element values ​​in () can be of the following three types: 1) When Key There is a vector key and Query When the correlation is high, the attention score between them Softmax () is close to 1, which will make: Softmax There are cases where one of the element values ​​in () is close to 1 and the rest are close to 0.

[0031] 2) When Key There are only a few in the vector key and Query When the correlation is high and close, Softmax The maximum value among the element values ​​of () will be Query Related key The increase in the number of log Type attenuation.

[0032] 3) When K All the ey vectors key and Query When the correlation is not high, Softmax The element values ​​of () are close to 1 / len ( Key ),in, len ( Key )express Key Length.

[0033] According to the above three situations: Softmax The element values ​​in () may have very different distributions, which is the main reason for the quantization loss of the Attention module. Figure 1 For a practical application scenario, Softmax Schematic diagram of the distribution of element values ​​of (), the X and Y axes represent Softmax The position of each element in (), the Z axis represents Softmax () in each element value. From the figure, we can see that: Softmax The distribution differences of the element values ​​of () are still quite large.

[0034] Figure 2 Flowchart of the attention feature extraction method provided by one embodiment of the present invention. Figure 2 As shown, the specific steps are as follows: Step 201: Obtain the content to be queried, the reference content, and the floating-point feature vector of the reference content.

[0035] The content to be queried refers to the content from which attention features are to be extracted. The reference content refers to the content related to the content to be queried. The floating-point feature vector of the reference content refers to the feature vector that has been extracted from the reference content, and the feature vector is of floating-point type.

[0036] The content types of the to-be-queried content and the reference content are the same, for example, the to-be-queried content and the reference content are both images, or both are texts, or both are voices, etc.

[0037] Step 202: Calculate a fixed-point attention weight vector of the content to be queried with respect to the reference content based on the content to be queried and the reference content.

[0038] In practical applications, this step 202 specifically includes: multiplying the fixed-point vector of the query content vector and the fixed-point vector of the transposed vector of the reference content vector by the square root of the dimension of the reference content vector, and using the fixed-point vector of the quotient vector as the fixed-point attention weight vector. For example: the fixed-point attention weight vector of the query content to the reference content P It can be expressed as: (3) in, Q Represents the content vector to be queried 、K represents the reference content vector, T is the transpose symbol, d express K The dimension of the vector, Quant () is the quantization function.

[0039] The attention weight vector of the query content to the reference content indicates the correlation between each part of the reference content and each part of the query content. The larger the weight in the attention weight vector, the greater the correlation between the part of the reference content corresponding to the weight and the corresponding part of the query content.

[0040] Step 203: The normalized result of the fixed-point attention weight vector is expressed as: the floating-point intermediate attention weight vector divided by the element sum of the floating-point intermediate attention weight vector.

[0041] Among them, each floating-point intermediate attention weight (i.e., each element) in the floating-point intermediate attention weight vector is represented by the following method: first, find the maximum value in the fixed-point attention weight vector; then calculate the difference between the fixed-point attention weight corresponding to the floating-point intermediate attention weight and the maximum value; finally, calculate eThe power with the difference as the base is the floating-point intermediate attention weight.

[0042] For example: Fixed-point attention weight vector P =[ p 1, p 2, …, p d ], d is a positive integer, then the fixed-point attention weight vector P The normalized result is expressed as:

[0043] in, Softmax () is the normalization function, Right now[ p 1, p 2, …, p d ], is the floating point intermediate attention weight vector, is the floating point intermediate attention weight vector M Elements and.

[0044] Step 204: quantize the floating-point intermediate attention weight vector, the floating-point feature vector of the reference content, and the element sum of the floating-point intermediate attention weight vector respectively to obtain a fixed-point value of the element sum of the fixed-point intermediate attention weight vector, the fixed-point feature vector of the reference content, and the floating-point intermediate attention weight vector.

[0045] Step 205: perform dot multiplication on the fixed-point intermediate attention weight vector and the fixed-point feature vector of the reference content, and then divide it by the fixed-point value of the sum of the elements of the floating-point intermediate attention weight vector, and use the fixed-point vector of the obtained quotient vector as the fixed-point attention feature vector of the content to be queried.

[0046] Steps 204-205 can be expressed as: Quant_Attn ( Q , K , V ) = Quant ( Quant ( M ) Quant ( V ) / Quant ( L )) (5) in, Q Represents the vector of the content to be queried, such as images, voice or text; KRepresents the reference content vector, such as image, speech or text; V for K The floating-point feature vector of ; Quant_Attn ( Q , K , V ) represents the fixed-point attention feature vector of the content to be queried; Quant () is the quantization function; M is the floating point intermediate attention weight vector; L is the floating point intermediate attention weight vector M Elements and.

[0047] From formula (4), we can see that: Softmax ( P ) is the maximum value difference after the division is completed, and the numerator M The maximum value is always 1, so Attn ( Q , K , V ) is transformed as follows to reduce the quantization error:

[0048] By Softmax The division operation " / L ”, put it M and V After multiplication, the problem of quantization dropout can be greatly alleviated. Under this transformation, the quantization Attn ( Q , K , V ) is formula (5): Quant_Attn ( Q , K , V ) = Quant ( Quant ( M ) Quant ( V ) / Quant ( L )) It can be seen that formula (5) does not introduce any new operations, all operations still maintain high-efficiency fixed-point operations, and the quantization loss is also reduced.

[0049] After obtaining the fixed-point attention feature vector of the content to be queried, the query task for the content to be queried can be performed based on the fixed-point attention feature vector of the content to be queried.

[0050] In an optional embodiment, when the content to be queried and the reference content are images, the query task is to query whether a certain target exists in the image to be queried, or to query the location of the target in the image to be queried. For example, query whether a tumor exists in the image to be queried, or to query the location of the tumor in the image to be queried, etc.

[0051] In an optional embodiment, when both the content to be queried and the reference content are texts, the query task is to translate the text to be queried into another language text, or to extract target summary information from the text to be queried. For example, to translate the text to be queried into English text, or to extract the object referred to by "it" from the text to be queried, etc.

[0052] In an optional embodiment, when both the content to be queried and the reference content are voices, the query task is: identifying the target information in the voice to be queried, or querying whether the voice to be queried is the voice of the target person. For example: identifying whether the tone of "shi" in the voice to be queried is the fourth tone or the light tone, or querying whether the voice to be queried is the voice of user A, etc.

[0053] In the above embodiment, the normalized result of the fixed-point attention weight vector is expressed as: the floating-point intermediate attention weight vector divided by the element sum of the floating-point intermediate attention weight vector, and then the floating-point intermediate attention weight vector, the floating-point feature vector of the reference content and the element sum of the floating-point intermediate attention weight vector are quantized respectively, and then the fixed-point intermediate attention weight vector and the fixed-point feature vector of the reference content are dot-multiplied and then divided by the fixed-point value of the element sum of the floating-point intermediate attention weight vector, and the fixed-point vector of the obtained quotient vector is used as the fixed-point attention feature vector of the content to be queried, thereby reducing the quantization loss in the attention feature extraction process without increasing the complexity of the attention feature extraction.

[0054] Figure 3 Schematic diagram of the structure of an attention feature extraction device provided by an embodiment of the present invention. Figure 3 As shown, the device mainly includes: a content acquisition module 31 and a feature extraction module 32, wherein: The content acquisition module 31 acquires the content to be queried, the reference content and the floating-point feature vector of the reference content.

[0055] The feature extraction module 32 calculates the fixed-point attention weight vector of the content to be queried for the reference content based on the content to be queried and the reference content; expresses the normalized result of the fixed-point attention weight vector as: the floating-point intermediate attention weight vector divided by the element sum of the floating-point intermediate attention weight vector; quantizes the floating-point intermediate attention weight vector, the floating-point feature vector of the reference content, and the element sum of the floating-point intermediate attention weight vector, respectively, to obtain the fixed-point value of the fixed-point intermediate attention weight vector, the fixed-point feature vector of the reference content, and the element sum of the floating-point intermediate attention weight vector; performs dot multiplication on the fixed-point intermediate attention weight vector and the fixed-point feature vector of the reference content, and then divides it by the fixed-point value of the element sum of the floating-point intermediate attention weight vector, and uses the fixed-point vector of the quotient vector as the fixed-point attention feature vector of the content to be queried.

[0056] In an optional embodiment, each floating-point intermediate attention weight in the floating-point intermediate attention weight vector in the feature extraction module 32 is obtained by: finding the maximum value in the fixed-point attention weight vector; calculating the difference between the fixed-point attention weight corresponding to the floating-point intermediate attention weight and the maximum value; calculating e The power with the difference as the base is the floating-point intermediate attention weight.

[0057] In an optional embodiment, the feature extraction module 32 calculates the fixed-point attention weight vector of the content to be queried to the reference content based on the content to be queried and the reference content, including: performing a dot multiplication on the fixed-point vector of the content vector to be queried and the fixed-point vector of the transposed vector of the reference content vector and then dividing by the square root of the dimension of the reference content vector, and using the fixed-point vector of the obtained quotient vector as the above-mentioned fixed-point attention weight vector.

[0058] In an optional embodiment, after the feature extraction module 32 uses the obtained fixed-point vector of the quotient vector as the fixed-point attention feature vector of the content to be queried, it further includes: executing a query task on the content to be queried according to the fixed-point attention feature vector of the content to be queried.

[0059] In an optional embodiment, the content to be queried and the reference content are images, and the feature extraction module 32 performs a query task on the content to be queried, including: querying whether a certain target exists in the image to be queried, or querying the position of the target in the image to be queried; Alternatively, the content to be queried and the reference content are both texts, and the feature extraction module 32 performs a query task on the content to be queried, including: translating the text to be queried into another type of language text, or extracting target summary information from the text to be queried; Alternatively, both the content to be queried and the reference content are voices, and the feature extraction module 32 performs a query task on the content to be queried, including: identifying target information in the voice to be queried, or querying whether the voice to be queried is the voice of a target person.

[0060] An embodiment of the present invention also provides a non-transitory computer-readable storage medium, which stores instructions that, when executed by a processor, enable the processor to perform the attention feature extraction method described in any of the aforementioned embodiments.

[0061] An embodiment of the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the attention feature extraction method described in any of the aforementioned embodiments.

[0062] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims disclosed herein may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present application. In particular, without departing from the spirit and teachings of the present application, the features described in the various embodiments and / or claims of the present application may be combined and / or combined in a variety of ways, and all of these combinations and / or combinations fall within the scope disclosed in the present application.

[0063] Specific embodiments are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas, and is not used to limit the present application. For those skilled in the art, changes can be made in the specific implementation methods and application scopes according to the ideas, spirits and principles of the present invention, and any modifications, equivalent substitutions, improvements, etc. made therein should be included in the scope of protection of this application.

Claims

1. A method for extracting attention features, characterized in that: The method includes: Obtaining the content to be queried, the reference content, and the floating-point feature vector of the reference content; According to the content to be queried and the reference content, a fixed-point attention weight vector of the content to be queried to the reference content is calculated; The normalized result of the fixed-point attention weight vector is expressed as: the floating-point intermediate attention weight vector divided by the element-wise sum of the floating-point intermediate attention weight vector; Quantize the floating-point intermediate attention weight vector, the floating-point feature vector of the reference content, and the element sum of the floating-point intermediate attention weight vector to obtain a fixed-point value of the element sum of the fixed-point intermediate attention weight vector, the fixed-point feature vector of the reference content, and the floating-point intermediate attention weight vector; The fixed-point intermediate attention weight vector is multiplied by the fixed-point feature vector of the reference content, and then divided by the fixed-point value of the sum of the elements of the floating-point intermediate attention weight vector, and the fixed-point vector of the obtained quotient vector is used as the fixed-point attention feature vector of the content to be queried.

2. The method according to claim 1, characterized in that: Each floating point intermediate attention weight in the floating point intermediate attention weight vector is obtained as follows: Find the maximum value in the fixed-point attention weight vector; Calculate the difference between the fixed-point attention weight corresponding to the floating-point intermediate attention weight and the maximum value; Calculate e The power with the difference as the base is the floating-point intermediate attention weight.

3. The method according to claim 1, characterized in that: The step of calculating the fixed-point attention weight vector of the content to be queried for the reference content according to the content to be queried and the reference content includes: The fixed-point vector of the content vector to be queried and the fixed-point vector of the transposed vector of the reference content vector are point-multiplied and then divided by the square root of the dimension of the reference content vector, and the fixed-point vector of the obtained quotient vector is used as the fixed-point attention weight vector.

4. The method according to claim 1, characterized in that After the obtained fixed-point vector of the quotient vector is used as the fixed-point attention feature vector of the content to be queried, the method further includes: According to the fixed-point attention feature vector of the content to be queried, the query task for the content to be queried is performed.

5. The method according to claim 4, characterized in that The to-be-queried content and the reference content are images; The query task is to query whether a certain target exists in the image to be queried, or to query the position of the target in the image to be queried.

6. The method according to claim 4, characterized in that The content to be queried and the reference content are both texts; The query task is to translate the text to be queried into another language text, or to extract target summary information from the text to be queried.

7. The method according to claim 4, characterized in that The to-be-queried content and the reference content are both voice; The query task is to identify target information in the voice to be queried, or to query whether the voice to be queried is the voice of a target person.

8. An attention feature extraction device, characterized in that: The device includes: A content acquisition module, which acquires the content to be queried, the reference content and the floating-point feature vector of the reference content; The feature extraction module calculates the fixed-point attention weight vector of the content to be queried for the reference content based on the content to be queried and the reference content; expresses the normalized result of the fixed-point attention weight vector as: the floating-point intermediate attention weight vector divided by the element sum of the floating-point intermediate attention weight vector; quantizes the floating-point intermediate attention weight vector, the floating-point feature vector of the reference content, and the element sum of the floating-point intermediate attention weight vector, respectively, to obtain the fixed-point value of the fixed-point intermediate attention weight vector, the fixed-point feature vector of the reference content, and the element sum of the floating-point intermediate attention weight vector; performs dot multiplication on the fixed-point intermediate attention weight vector and the fixed-point feature vector of the reference content, and then divides it by the fixed-point value of the element sum of the floating-point intermediate attention weight vector, and uses the fixed-point vector of the quotient vector as the fixed-point attention feature vector of the content to be queried.

9. The device according to claim 8, characterized in that Each floating-point intermediate attention weight in the floating-point intermediate attention weight vector in the feature extraction module is obtained as follows: Find the maximum value in the fixed-point attention weight vector; Calculate the difference between the fixed-point attention weight corresponding to the floating-point intermediate attention weight and the maximum value; Calculate e The power with the difference as the base is the floating-point intermediate attention weight.

10. The device according to claim 8, characterized in that After the feature extraction module uses the obtained fixed-point vector of the quotient vector as the fixed-point attention feature vector of the content to be queried, the feature extraction module further includes: According to the fixed-point attention feature vector of the content to be queried, the query task for the content to be queried is performed.

11. The device according to claim 10, characterized in that The content to be queried and the reference content are images, and the feature extraction module performs a query task on the content to be queried, including: querying whether a certain target exists in the image to be queried, or querying the position of the target in the image to be queried; Alternatively, the content to be queried and the reference content are both texts, and the feature extraction module performs a query task on the content to be queried, including: translating the text to be queried into another type of language text, or extracting target summary information from the text to be queried; Alternatively, the content to be queried and the reference content are both voices, and the feature extraction module performs a query task on the content to be queried, including: identifying target information in the voice to be queried, or querying whether the voice to be queried is the voice of a target person.

12. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores instructions, which, when executed by a processor, cause the processor to perform the attention feature extraction method as described in any one of claims 1 to 7.

Images