Attention module and data processing system
By introducing an attention module into the data processing system, features are first filtered according to the query purpose, and then the features are integrated. This solves the problem of redundant information in existing technologies, optimizes computing resources, and improves the accuracy of the information of interest.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-29
- Publication Date
- 2026-03-31
AI Technical Summary
Existing data processing systems suffer from excessive redundancy in the process of extracting attention information, leading to high computational resource consumption.
By introducing an attention module into the data processing system, features in the corresponding dimensions are first filtered according to the query purpose, and then the input features are multiplied by the filtered dimensions to integrate the features and reduce redundant information in the neural network structure.
It reduces the overhead of computing resources and improves the accuracy of the attention module in extracting information of interest.
Smart Images
Figure CN121765331A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and deep learning, and more particularly to an attention module and a data processing system. Background Technology
[0002] With advancements in research on visual attention mechanisms, the process of generating attention information from incoming stimuli has been abstracted into different attention models. From Broadbent's 1958 proposal of filtering and then abstracting features, to Treisman and Gelade's 1980 feature integration theory, the structural approaches to abstracting visual features and attention models have been updated. Since integration theory is a crucial cornerstone of Load Theory and Dilution Theory, and an important component of Hybrid Theory, it provides a theoretical basis for explaining the different experimental results in the debate between strictly pre-selective and post-selective attention mechanisms. However, existing data processing systems often employ a method of first extracting all abstract features, multiplying them to form a new feature matrix, and then performing feature selection. This results in a large amount of redundant information during feature selection, significantly increasing the computational demands of the data processing system and raising the computational resource overhead of the attention model.
[0003] In order to overcome the above-mentioned defects of the existing technology, there is an urgent need in the field for an improved attention module to reduce the amount of redundant information in the neural network structure, thereby reducing the computational resource system overhead. Summary of the Invention
[0004] The following provides a brief overview of one or more aspects to offer a basic understanding of them. This overview is not an exhaustive summary of all conceived aspects, nor is it intended to identify key or decisive elements of all aspects, nor to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed descriptions that follow.
[0005] To overcome the aforementioned deficiencies in the existing technology, this invention provides an attention module and a data processing system. This system can first filter features in the corresponding dimension according to the query purpose, and then multiply the input features in the filtered corresponding dimension to perform feature integration. This is used to concentrate computing resources on specific information elements that meet the query purpose, thereby reducing the amount of redundant information in the neural network structure and thus reducing the computing resource system overhead.
[0006] Specifically, the attention module provided according to the first aspect of the present invention includes an input feature generation unit, a context feature generation unit, a filtering unit, and an attention information extraction unit. The input feature generation unit generates input features of a first dimension based on keys and values at multiple locations in the input data. The context feature generation unit generates context features based on keys at multiple locations in the input data. The filtering unit includes a bandwidth matrix indicating the purpose of querying the input data, used to filter the context features to generate an attention weight matrix. The attention information extraction unit generates attention information of a second dimension based on the dot product of the attention weight matrix and the input features. The second dimension is smaller than the first dimension.
[0007] Furthermore, in some embodiments of the present invention, the attention module is also configured with a query matrix. The query matrix has the same third dimension as the bandwidth matrix. The context feature generation unit is configured to generate a fourth-dimensional context feature based on keys at multiple positions in the input data and the query matrix. The fourth dimension is less than the first dimension and greater than the third dimension.
[0008] Furthermore, in some embodiments of the present invention, the filtering unit is configured to: acquire filter features indicating the query purpose of the input data; and, based on the filter features, set the dimensions in the query matrix that do not conform to the query purpose to zero, so as to generate the bandwidth matrix.
[0009] Furthermore, in some embodiments of the present invention, the step of obtaining filter features indicating the query purpose of the input data includes: obtaining feedforward information and / or feedback information output by the neural network module behind the attention module; and generating the filter features based on the feedforward information and / or the feedback information.
[0010] Furthermore, in some embodiments of the present invention, the filtering unit is further configured to: calculate a first dot product between the bandwidth matrix and the context features; and normalize the first dot product to generate the fourth-dimensional attention weight matrix.
[0011] Furthermore, in some embodiments of the present invention, the attention information extraction unit is configured to: generate multiple candidate attention information of different dimensions by reducing the dimensionality based on the dot product of the attention weight matrix and the input features; and select the candidate attention information with the highest attention from each candidate attention information as the attention information of the second dimension.
[0012] Furthermore, the data processing system provided according to a second aspect of the present invention includes an attention module and at least one post-processing module. The attention module is used to extract attention information that conforms to the query objective from the input data to be processed. The at least one post-processing module is used to output a data processing result that conforms to the query objective based on the attention information.
[0013] Furthermore, in some embodiments of the present invention, the at least one post-processing module includes an addition module, a feedforward neural network module, and a first normalization module. The addition module is used to superimpose the input features of the attention module and its output attention information to form a fifth-dimensional post-processing feature. The fifth dimension is the sum of the first dimension and the second dimension. The feedforward neural network module is used to enhance the positional information in the post-processing feature through mapping of hidden layers. The first normalization module is used to generate the data processing result based on the feedforward information output by the feedforward neural network module.
[0014] Furthermore, in some embodiments of the present invention, the data processing system is implemented based on a sequence model, and further includes a data mapping module, a position encoding module, a multi-level encoder, and a multi-level decoder. The data mapping module is used to map the input data to be processed into multiple fixed-length input vectors according to a preset data unit based on a pre-constructed embedding matrix, and concatenate the multiple fixed-length input vectors into an input embedding. Each dimension in the input embedding includes a unique key and a corresponding value. The position encoding module is used to determine a corresponding position code based on the position of each dimension in the input embedding, and generate input features based on the key, value, and position code of each dimension in the input embedding. The multi-level encoder and multi-level decoder are respectively configured with the attention module, the addition module, and a second normalization module. The second normalization module is used to normalize the post-processed features output by the addition module to the first dimension and input it into the attention module of the next encoder or decoder. The addition module of the last level decoder directly outputs its post-processed features to the feedforward neural network module.
[0015] Furthermore, in some embodiments of the present invention, the input data to be processed is selected from image data or text data. The data processing result is correspondingly selected from image classification labels, target recognition labels, or semantic labels. Attached Figure Description
[0016] The above-described features and advantages of the present invention will be better understood after reading the following detailed description of embodiments of the present disclosure in conjunction with the accompanying drawings. In the drawings, components are not necessarily drawn to scale, and components having similar related characteristics or features may have the same or similar reference numerals.
[0017] Figure 1 A schematic diagram illustrating the mapping of neuroscience principles to computer science principles according to some embodiments of the present invention is shown.
[0018] Figure 2 A schematic diagram illustrating the conceptual organization involved in a model of attention mechanisms in neuroscience provided according to some embodiments of the present invention is shown.
[0019] Figure 3 A schematic diagram illustrating the conceptual organization involved in the attention mechanism model in computer science provided according to some embodiments of the present invention is shown.
[0020] Figure 4 A schematic diagram of the structure of an attention module provided according to some embodiments of the present invention is shown.
[0021] Figure 5 A schematic diagram illustrating the principle of extracting information of interest according to some embodiments of the present invention is shown.
[0022] Figure 6 A schematic diagram of the structure of a data processing system provided according to some embodiments of the present invention is shown. Detailed Implementation
[0023] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Although the description of the present invention is presented in conjunction with preferred embodiments, this does not mean that the features of the invention are limited to these embodiments. On the contrary, the purpose of describing the invention in conjunction with embodiments is to cover other options or modifications that may be derived based on the claims of the present invention. To provide a thorough understanding of the invention, many specific details will be included in the following description. The invention may also be implemented without using these details. Furthermore, to avoid confusion or obscuring the focus of the invention, some specific details will be omitted in the description.
[0024] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.
[0025] Furthermore, the terms "upper," "lower," "left," "right," "top," "bottom," "horizontal," and "vertical" used in the following description should be understood as the orientations shown in the relevant paragraphs and accompanying drawings. These relative terms are for illustrative purposes only and do not imply that the described apparatus must be manufactured or operated in a specific orientation, and therefore should not be construed as limiting the invention.
[0026] It is understood that although terms such as "first," "second," and "third" may be used herein to describe various components, regions, layers, and / or parts, these components, regions, layers, and / or parts should not be limited by these terms, and these terms are only used to distinguish different components, regions, layers, and / or parts. Therefore, the first components, regions, layers, and / or parts discussed below may be referred to as second components, regions, layers, and / or parts without departing from some embodiments of the present invention.
[0027] As mentioned above, existing data processing systems extract attention information by first multiplying all abstract features to form a new feature matrix, and then performing feature selection. This results in a large amount of redundant information during feature selection, significantly increasing the computational resource overhead of the Attention model.
[0028] To overcome the aforementioned deficiencies in the existing technology, this invention provides an attention module and a data processing system. This system can first filter features in the corresponding dimension according to the query purpose, and then multiply the input features in the filtered corresponding dimension to perform feature integration. This is used to concentrate computing resources on specific information elements that meet the query purpose, thereby reducing the amount of redundant information in the neural network structure and thus reducing the computing resource system overhead.
[0029] Please refer to the reference. Figures 1-3 . Figure 1 A schematic diagram illustrating the mapping of neuroscience principles to computer science principles according to some embodiments of the present invention is shown. Figure 2 A schematic diagram illustrating the conceptual organization involved in a model of attention mechanisms in neuroscience provided according to some embodiments of the present invention is shown. Figure 3 A schematic diagram illustrating the conceptual organization involved in the attention mechanism model in computer science provided according to some embodiments of the present invention is shown.
[0030] like Figure 1As shown, the data processing system provided by the second aspect of the present invention can map the process of generating attention information from stimulus signals in brain science research to the algorithmic implementation of attention models in computer science. Specifically, the brain's abilities such as consciousness, thinking, understanding, and behavior can be mapped to the understanding and reaction performance of the data processing system. During the brain's visual perception function of processing image or text data, the brain first selectively attends to image or text features and then summarizes the features. For information that does not require attention, the brain automatically filters it with a certain inhibition rate. Similarly, during the image classification function of processing image data, the data processing system can mimic the brain's behavior, first filtering features of the corresponding dimension according to the adaptive attention query purpose, and using different image masks to identify different regions in the image for further filtering. In this way, the modular encoding of the brain's visual features can be mapped to the pre-selected feature mimicry of the data processing system, thereby reducing the computational resource overhead.
[0031] like Figure 2 As shown, in the field of neuroscience, the conceptual organization involved in attention mechanism models mainly involves three categories of concepts: ability, function, and resources. Here, consciousness, understanding, thinking, and reaction belong to multiple categories of brain abilities. Perception is a category of brain function, its specific forms including vocabulary, vision, smell, and touch. Cognition is also a category of brain function, including attention, executive function, and memory. The various regions and networks within the brain belong to the brain's resources.
[0032] Accordingly, such as Figure 3 As shown, in the field of computer science, the concept organization of attention mechanism models in corresponding neuroscience mainly involves three categories of concepts: ability, function, and resources. Here, computers also possess abilities such as consciousness and reaction, and functions such as perception and cognition. Specific forms of perception include vocabulary and vision, while cognition includes the computer's attention with feature-selective gain. The computer's resources are random inhibitory neurons in feedback and feedforward neural networks.
[0033] Furthermore, in some non-limiting embodiments, the attention module provided in the first aspect of the present invention can be configured to be implemented in the data processing system provided in the second aspect of the present invention.
[0034] Specifically, the data processing system provided in the second aspect of the present invention includes the attention module 11 and at least one post-processing module provided in the first aspect of the present invention. The attention module 11 is used to extract attention information that meets the query purpose from the input data to be processed. The post-processing module is used to output data processing results that meet the query purpose based on the attention information. Here, the input data to be processed is selected from image data or text data, and the data processing results are correspondingly selected from image classification labels, target recognition labels, or semantic labels.
[0035] Please refer to the reference for details. Figure 4 and Figure 5 . Figure 4 A schematic diagram of the structure of an attention module provided according to some embodiments of the present invention is shown. Figure 5 A schematic diagram illustrating the principle of extracting information of interest according to some embodiments of the present invention is shown.
[0036] exist Figure 4 In the illustrated embodiment, the attention module 11 provided in the first aspect of the present invention includes an input feature generation unit, a context feature generation unit, a filtering unit, and an attention information extraction unit. Here, the input feature generation unit is used to generate input features I of a first dimension k*v based on keys and values at multiple positions in the input data. k*v This context feature generation unit is used to generate context features C based on keys at multiple locations in the input data. k*q The filtering unit includes a bandwidth matrix that indicates the query purpose of the input data, used to filter contextual features C. k*q This is used to generate attention weight matrices. The attention information extraction unit is used to extract attention weight matrices based on the input features I. k*v The dot product of these values generates the second dimension of Attended Information. Here, the second dimension is smaller than the first dimension k*v.
[0037] Furthermore, in some preferred embodiments, the attention module 11 provided in the first aspect of the present invention is also configured with a query matrix. Here, the query matrix includes query vectors corresponding to keys and values. The query matrix and the bandwidth matrix have the same third dimension q*q. Specifically, in embodiments configured with a query matrix, the aforementioned context feature generation unit can generate a fourth dimension k*q context feature C based on keys at multiple positions in the input data and the query matrix. k*q Here, the fourth dimension k*q is less than the first dimension k*v, and greater than the third dimension q*q.
[0038] Furthermore, in some optional embodiments, the filtering unit may calculate the query matrix and context features C. k*q The dot product of these values is used to obtain the corresponding attention map, which is then normalized (Softmax) to directly generate the fourth-dimensional k*q attention weight matrix.
[0039]
[0040] Among them, Q i For the i-th query vector in the query matrix, Input feature I k*v The transpose of the j-th key vector, d k is the dimension of the key vector.
[0041] In addition, Figure 4 In the illustrated embodiment, the filtering unit can acquire feedforward and / or feedback information output by the neural network module behind the attention module 11, and generate filter features F indicating the query purpose of the input data based on the feedforward and / or feedback information. q*q And based on filter feature F q*q The dimensions in the query matrix that do not meet the query objective are set to zero to generate a bandwidth matrix.
[0042] Specifically, for the filter features F with q dimensions q*q The filtering unit can determine the bandwidth zeroing dimension parameter based on feedforward and / or feedback information. Here, the bandwidth zeroing dimension parameter is greater than or equal to 1, meaning that by default, at least one dimension of the query matrix has data zeroed to form the bandwidth matrix. For example, if the bandwidth zeroing dimension parameter is set to 3, the filtering unit can zero out 6 elements in the upper triangle of dimension 3 and 6 elements in the lower triangle of dimension 3 in the query matrix to generate the bandwidth matrix.
[0043] Furthermore, in some preferred embodiments, the filtering unit can also calculate the bandwidth matrix and context features C. k*q The dot product of the numbers is then normalized to generate a fourth-dimensional k*q attention weight matrix:
[0044]
[0045] Among them, B i Let be the i-th bandwidth vector in the bandwidth matrix.
[0046] Here, the bandwidth matrix has more zero values than the query matrix, significantly reducing the computational complexity of the data processing system. Thus, the attention module 11 provided in the first aspect of the present invention can, via the filtering unit, process the context feature C. k*q Filtering is performed to concentrate computational resources on specific information elements that match the query objective, thereby reducing the number of input features and thus lowering the computational resource overhead. Furthermore, since the filtered features are more targeted, the accuracy of the attention module 11 in extracting relevant information can be improved.
[0047] Furthermore, in some preferred embodiments, the attention information extraction unit can extract information based on the attention weight matrix and the input features I. k*v The dot product is used to reduce the dimensionality and generate candidate attention information of various dimensions. Here, the various dimensions are k*(q / n), where n is an integer greater than 1 and divisible by q. Then, the attention information extraction unit can select the one with the highest attention from each candidate attention information according to the maximum, median, or average value of each dimension, as the attention information of the second dimension.
[0048] Thus, the attention module 11 can further reduce the dimensionality of the attention information and reduce the computational resource system overhead by further filtering from each candidate attention information according to different dimensions.
[0049] like Figure 5 As shown, during the calculation of the attention weight matrix, the filtering unit can use the filter features F with the third dimension q*q. q*q The filter unit selects features by setting dimensions in the query matrix that do not meet the query objective to zero, thus generating a bandwidth matrix. Afterward, the filter unit can calculate the contextual feature C of the fourth dimension k*q. k*q The attention weight matrix is generated by taking a dot product with the bandwidth matrix and then normalizing the dot product to produce a fourth-dimensional k*q attention weight matrix. Afterwards, the attention information extraction unit can use the attention weight matrix and the input feature I... k*v The dot product is used to reduce the dimensionality and generate candidate attention information of various dimensions. Then, the attention information extraction unit can select the one with the highest attention from each candidate attention information according to the maximum, median or average value of each dimension, as the attention information of the second dimension.
[0050] Please refer to Figure 6 . Figure 6 A schematic diagram of the structure of a data processing system provided according to some embodiments of the present invention is shown.
[0051] Furthermore, in Figure 6In the illustrated embodiment, the post-processing module 12 in the data processing system provided by the second aspect of the present invention may include an addition module 121, a feedforward neural network module 122, and a first normalization module 123. The addition module 121 is used to superimpose the input features of the attention module 11 and its output attention information to form a post-processed feature of a fifth dimension. Here, the fifth dimension can be the sum of the first dimension k*v and the second dimension. The feedforward neural network module 122 is used to enhance the positional information in the post-processed feature through the mapping of the hidden layers.
[0052] FFN(x)=ReLU(xW1+b1)W2+b2
[0053] Where W1 and W2 are weight matrices, and b1 and b2 are bias vectors.
[0054] The first normalization module 123 is used to generate data processing results based on the feedforward information output by the feedforward neural network module 122.
[0055] In addition, Figure 6 In the illustrated embodiment, the data processing system provided by the second aspect of the present invention is implemented based on a sequence model. This sequence model further includes a data mapping module, a position encoding module, a multi-level encoder, and a multi-level decoder.
[0056] Specifically, this data mapping module maps the input data to be processed into multiple fixed-length input vectors according to a pre-constructed embedding matrix, using preset data units (e.g., a pixel, a pixel matrix of a specific size, or a word). These fixed-length input vectors are then concatenated to form an input embedding. Each dimension of the input embedding includes a unique key and a corresponding value. This embedding matrix is obtained through pre-training. In this way, the data mapping module can transform discrete input data into continuous vector representations that can be directly input into a neural network, enabling the model to process and understand the input data.
[0057] Furthermore, in some preferred embodiments, each dimension of the input embedding may also include a query vector corresponding to the key and value. Here, the input embedding vector can be mapped to the query vector Q through three linear transformations. i Key K i Sum V i :
[0058] Q i =WQ ·x i
[0059] K i =W K ·x i
[0060] V i =W V ·x i
[0061] Among them, W Q W K and W V These are learnable weight matrices for the query vector, key, and value, respectively.
[0062] The position encoding module 13 is used to determine a corresponding position code based on the position of each dimension in the input embedding, and to generate input features based on the key, value, and position code of each dimension in the input embedding. Here, the position code is typically generated using sine and cosine functions.
[0063] For example, for the 2i-th dimension in the input embedding, the position encoding of position pos is:
[0064]
[0065] For example, for the (2i+1)th dimension of the input embedding, the position encoding of position pos is:
[0066]
[0067] Where pos is the position index, i is the dimension index, and d is the position index. model The dimension of the input embedding vector.
[0068] The multi-level encoder and multi-level decoder are respectively configured with an attention module 11, an addition module 121, and a second normalization module 124. Here, the second normalization module 124 is used to normalize the post-processed features output by the addition module 121 to a first dimension k*v, and input it into the attention module 11 of the next encoder or decoder. The addition module of the last level decoder directly outputs its post-processed features to the feedforward neural network module 122.
[0069] In addition, Figure 6 In the illustrated embodiment, the sequence model further includes a linear processing module 14. Here, the linear processing module 14 is used to generate output representations for downstream tasks such as classification labels or object detection results based on the post-processing features output by the last-stage decoder.
[0070] Furthermore, in some alternative embodiments, the data processing system described above in the second aspect of the present invention can also achieve similar functionality based on a CNN model with attention module 11. Here, the attention module 11 provided in the first aspect of the present invention can be used to replace the Attention model in the CNN model, for extracting attention information that meets the query purpose from the input data to be processed.
[0071] Furthermore, in some alternative embodiments, the data processing system described above provided in the second aspect of the present invention can also be used for basic data processing functions such as image classification, target recognition, and semantic segmentation.
[0072] For example, in an image classification task, the data processing system can first process the input image through a data mapping module and a location encoding module 13 to obtain keys and values at multiple locations in the input image, as well as the corresponding query matrix. Then, the data processing system can generate second-dimensional attention information through an attention module 11. Next, the data processing system can continuously superimpose post-processed features and attention information through a multi-level encoder and a multi-level decoder, normalize them, and then input them into a feedforward neural network module 122. Finally, the data processing module can map the attention information to image classification labels and iteratively optimize the mapped attention weight matrix to obtain an image classifier, thereby classifying the input image.
[0073] For example, in an object detection task, the data processing system can first process the input image to be identified via a data mapping module and a location encoding module 13 to obtain the keys and values at multiple locations in the image, as well as the corresponding query matrix. Then, the data processing system can generate second-dimensional attention information via an attention module 11. Next, the data processing system can continuously superimpose post-processed features and attention information via a multi-level encoder and a multi-level decoder, normalize them, and then input them into a feedforward neural network module 122. Finally, the data processing module can map the attention information to object recognition labels and iteratively optimize the mapped attention weight matrix to generate the location information and category of the target to be identified, thereby accurately identifying the target in the image, such as a vehicle or pedestrian.
[0074] For example, in a semantic segmentation task, the data processing system can first process the input text data through a data mapping module and a position encoding module 13 to obtain the keys and values at multiple positions in the text data, as well as the corresponding query matrix. Then, the data processing system can generate second-dimensional attention information through an attention module 11. Next, the data processing system can continuously superimpose post-processed features and attention information through a multi-level encoder and a multi-level decoder, normalize them, and then input them into a feedforward neural network module 122. Finally, the data processing module can map the attention information to specific semantic labels and iteratively optimize the mapping attention weight matrix to generate the final text semantic segmentation result. This result is used to label each word as a corresponding category, enabling the system to accurately identify the grammatical and semantic structures in the text.
[0075] Furthermore, in some alternative embodiments, the data processing system described above provided in the second aspect of the present invention can also be further applied to video understanding, medical diagnosis, and autonomous driving based on image classification results, object recognition results, and semantic segmentation results.
[0076] In summary, the attention module and data processing system provided by this invention can integrate features by first filtering features in the corresponding dimension according to the query purpose, and then multiplying the input features in the filtered corresponding dimension. This concentrates computing resources on specific information elements that meet the query purpose, thereby reducing the amount of redundant information in the neural network structure and thus reducing the overhead of computing resources.
[0077] Although the methods described above are illustrated and depicted as a series of actions for the sake of simplicity, it should be understood and appreciated that these methods are not limited by the order of the actions, as some actions may occur in a different order and / or concurrently with other actions from the illustrations and descriptions herein or not illustrated and described herein but which may be understood by those skilled in the art, according to one or more embodiments.
[0078] Those skilled in the art will understand that information, signals, and data can be represented using any of a variety of different techniques and arts. For example, the data, instructions, commands, information, signals, bits, symbols, and chips described throughout the above description can be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or optical particles, or any combination thereof.
[0079] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps are described above in a generalized manner in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the invention.
[0080] The prior description of this disclosure is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not intended to be limited to the examples and designs described herein, but should be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. An attention module, characterized in that, The attention module comprises: an input feature generation unit configured to generate input features of a first dimension according to keys and values of a plurality of positions in input data; a context feature generation unit configured to generate context features according to keys of the plurality of positions in the input data; a filtering unit comprising a bandwidth matrix indicating a query purpose for the input data, and configured to filter the context features to generate an attention weight matrix; and an attention information extraction unit configured to generate attention information of a second dimension according to a dot product of the attention weight matrix and the input features, wherein the second dimension is smaller than the first dimension. The attention module further comprises a query matrix having a same third dimension as the bandwidth matrix, and the context feature generation unit is configured to:
2. The attention module of claim 1, wherein, generate context features of a fourth dimension according to the keys of the plurality of positions in the input data and the query matrix, wherein the fourth dimension is smaller than the first dimension and larger than the third dimension. The filtering unit is configured to:
3. The attention module of claim 2, wherein, obtain filter features indicating the query purpose for the input data; and zero out dimensions in the query matrix that do not meet the query purpose according to the filter features to generate the bandwidth matrix. The obtaining of the filter features comprises:
4. The attention module of claim 3, wherein, obtaining feedforward information and / or feedback information output by a neural network module at a backend of the attention module; and generating the filter features according to the feedforward information and / or the feedback information. The filtering unit is further configured to:
5. The attention module of claim 3, wherein, calculate a first dot product of the bandwidth matrix and the context features; and normalize the first dot product to generate the attention weight matrix of the fourth dimension. The attention information extraction unit is configured to:
6. The attention module of claim 1, wherein, generate candidate attention information of a plurality of different dimensions according to a dot product of the attention weight matrix and the input features; and select, from the candidate attention information, one having a highest attention degree as the attention information of the second dimension. The data processing system comprises:
7. A data processing system, characterized by the attention module according to any one of claims 1-6, configured to extract attention information meeting a query purpose from input data to be processed; and at least one post-processing module configured to output a data processing result meeting the query purpose according to the attention information. The at least one post-processing module comprises: an addition module configured to add input features of the attention module and the attention information output by the attention module to generate post-processing features of a fifth dimension, wherein the fifth dimension is a sum of the first dimension and the second dimension; 8. The data processing system of claim 7, wherein, a feedforward neural network module configured to strengthen position information in the post-processing features through mapping of a hidden layer; and a first normalization module configured to generate the data processing result according to feedforward information output by the feedforward neural network module. The data processing system is implemented based on a sequence model, and further comprises: 9. The data processing system of claim 8, wherein, The data mapping module is configured to map the input data to be processed into a plurality of fixed-length input vectors according to a preset data unit based on a pre-constructed embedding matrix, and splice the plurality of fixed-length input vectors into an input embedding, wherein each dimension in the input embedding includes a unique key and a corresponding value. The position encoding module is configured to determine a corresponding position encoding according to the position of each dimension in the input embedding, and generate input features according to the keys, values and position encodings of the dimensions in the input embedding. The multi-level encoder and the multi-level decoder are respectively configured with the attention module, the addition module and the second normalization module, wherein the second normalization module is configured to normalize the post-processing features output by the addition module to the first dimension, and input the post-processing features into the attention module of the next encoder or decoder, and the addition module of the last decoder directly outputs the post-processing features to the feedforward neural network module.
10. The data processing system of claim 7, wherein, The input data to be processed is selected from image data or text data, and the data processing result is correspondingly selected from image classification labels, target recognition labels or semantic labels.