Feature extraction method and device based on improved attention and dynamic convolution

Abstract

The invention discloses a feature extraction method and device based on improved attention and dynamic convolution. The method comprises the following steps: acquiring image input data of a to-be-extracted feature object; respectively transmitting the image input data to a first channel based on attention improvement and a second channel based on dynamic convolution; and generating attention mapping based on the integration area through the first channel, generating weight and value mapping through the second channel, obtaining image output data of the object to be subjected to feature extraction based on the integration area, the weight and the value mapping, and completing feature extraction. According to the method, different features are extracted at different positions by using the dynamically generated content related weights, so that the feature extraction efficiency is improved under the condition of fixed calculation cost.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically to a feature extraction method and apparatus based on improved attention and dynamic convolution. Background Technology

[0002] In recent years, learning frameworks based on convolutional neural networks (CNNs) have achieved great success in a range of tasks, including classification, object detection, and segmentation. Significant progress has been made in improving CNN performance by modifying the overall architecture. On the other hand, given the remarkable performance of self-attention mechanisms in natural language processing (NLP), some recent research has attempted to introduce attention-based mechanisms to modify standard convolutional blocks. In short, for a given location, a self-attention mechanism constructs the neighborhood interactions of that location by computing the attention values ​​of that location with respect to all its neighbors. The attention map, representing the set of attention values, is typically constructed from key and query matrices, both of which are linear projections of the input. These attention mechanisms, integrated with CNNs, outperform state-of-the-art CNN-based networks and are an important addition to the visual practitioner's toolkit.

[0003] Based on the size of neighboring locations, these self-attention and CNN ensemble methods can be divided into two categories: The first category is based on global attention methods, where the entire feature map should be considered during the computation of neighborhood interactions at a specific location. In existing methods, the neighborhood of a specific location is defined as the entire feature map, thus memory footprint and computational cost increase quadratically with the input size. The second category is based on local attention methods, which can reduce memory footprint and computational complexity; however, this method only involves neighbors within a small surrounding area in the computation.

[0004] Most self-attention and CNN ensemble methods suffer from a trade-off between computational complexity and the effectiveness of capturing long-range dependencies. Furthermore, traditional convolutional operations typically use a constant kernel at different locations, resulting in the extraction of identical features across different positions, thus reducing feature extraction efficiency. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes a feature extraction method and apparatus based on improved attention and dynamic convolution. This invention simultaneously considers the interaction between the input content and the convolution operation itself. The proposed convolutional block can dynamically change according to the input content, thereby effectively extracting long-range dependencies and local features.

[0006] To achieve the above objectives, the present invention provides the following solution: In a first aspect, the present invention proposes a feature extraction method based on improved attention and dynamic convolution, the method comprising the following steps: Obtain the image input data of the object whose features are to be extracted; The image input data is fed into a first channel based on improved attention and a second channel based on dynamic convolution, respectively. Generate an integrated region through the first channel. attention mapping Weights are generated through the second channel. Sum value mapping Based on the integrated region Weight and value mapping Obtain the output data of the feature object to be extracted , represented as: ; Feature extraction is completed using the output data of the object to be extracted obtained above.

[0007] Furthermore, the first channel generates a data based on the integrated region. attention mapping The specific steps include: The attention map A is calculated based on the key K and query Q, and is represented as follows: ; After transforming the above equation, we get: ; Will Set to a positive semi-definite matrix, then the attention mapping... Represented as: ; in, and This represents the parameter matrix based on a linear mapping. Represented as a four-dimensional matrix, Indexes representing different dimensions. It is a positive semi-definite matrix. The image input features represent the position (i, j). The image input features represent the location (a, b).

[0008] Furthermore, the first channel generates a data based on the integrated region. attention mapping The steps also include: In a specific location Several expansion options D are defined in the convolutional block, and these expansion options D correspond to different receptive field sizes. For any expansion option, the expansion coefficient is... The corresponding area The weights generated through the second channel Regional integration was carried out to obtain the integrated region. , represented as: ; in, express In element.

[0009] Furthermore, the expansion coefficient for any expansion option and its corresponding region In the middle, the corresponding receptive field size is .

[0010] Furthermore, weights are generated through the second channel. Sum value mapping The specific steps include: Applying convolution kernels to the image input data W O To calculate option mapping O And it is expressed as follows: ; According to a specific location Dynamically generate projection matrix of content And it is expressed as follows: ; Then at a specific location Calculate the value mapping at the location and according to the mapping Get value mapping , : ; in, express The The element at position 1 The parameter set corresponding to the convolution operation itself. W represents the weight set matrix. i This indicates a predefined kernel.

[0011] Furthermore, the region The size is 9.

[0012] Furthermore, the integrated region Including star-shaped areas The star-shaped region Represented as: ; Among them, the star-shaped region The size is represented as: .

[0013] In a second aspect, the present invention also proposes a feature extraction apparatus based on improved attention and dynamic convolution, the feature extraction apparatus performing the feature extraction method as described in the first aspect, the apparatus comprising: The acquisition module is used to acquire image input data of the object whose features are to be extracted; The processing module is used to send the image input data to a first channel based on improved attention and a second channel based on dynamic convolution, respectively; The privilege extraction module is used to generate a data structure based on the integrated region through the first channel. attention mapping Weights are generated through the second channel. Sum value mapping Based on the integrated region Weight and value mapping Obtain the output data of the feature object to be extracted , represented as: ; Feature extraction is completed using the output data of the object to be extracted obtained above.

[0014] Thirdly, the present invention also proposes an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the feature extraction method based on improved attention and dynamic convolution as described in the first aspect.

[0015] Fourthly, the present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the feature extraction method based on improved attention and dynamic convolution as described in the first aspect.

[0016] Compared with existing attention-based convolutional blocks, this invention has the following significant advantages and positive effects: (1) More effective extraction of long-range dependencies and local features: A novel content-dependent convolutional block is proposed, which considers the interaction between the input content and the convolution operation itself. The proposed convolutional block can dynamically change according to the input content, thereby effectively extracting long-range dependencies and local features.

[0017] (2) Relatively constant computational complexity: In order to balance the trade-off between computational complexity and neighborhood size, this invention proposes a new attention mechanism that captures long-range dependencies by adding several extended convolutions to the computational value graph. The integration of this extended convolution with the attention mechanism can theoretically enrich the foresight of traditional attention mechanisms while maintaining relatively constant computational complexity.

[0018] (3) Higher feature extraction efficiency: This invention proposes a new dynamic convolution method, which uses dynamically generated content-related weights to extract different features at different positions, thereby improving feature extraction efficiency with a fixed computational cost. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of an overall embodiment of the feature extraction method based on improved attention and dynamic convolution proposed in this invention.

[0020] Figure 2 This is a schematic diagram of integrated region generation in the feature extraction method based on improved attention and dynamic convolution proposed in this invention.

[0021] Figure 3 This is a schematic diagram of a specific embodiment of the feature extraction method based on improved attention and dynamic convolution proposed in this invention. Detailed Implementation

[0022] The present application will now be described in further detail with reference to the accompanying drawings. It should be noted that the following specific embodiments are only used to further illustrate the present application and should not be construed as limiting the scope of protection of the present application. Those skilled in the art can make some non-essential improvements and adjustments to the present application based on the above application content.

[0023] like Figure 1 As shown, this invention proposes a feature extraction method based on improved attention and dynamic convolution, the method comprising: Obtain the image input data of the object whose features are to be extracted; The image input data is fed into a first channel based on improved attention and a second channel based on dynamic convolution, respectively. Generate an integrated region through the first channel. attention mapping Weights are generated through the second channel. Sum value mapping Based on the integrated region Weight and value mapping Obtain the output data of the feature object to be extracted , represented as: ; Feature extraction is completed using the output data of the object to be extracted obtained above.

[0024] In this embodiment, the proposed feature extraction method based on improved attention and dynamic convolution generates a feature based on the integration region through the first channel. attention mapping The specific steps include: The attention map A is calculated based on the key K and query Q, and is represented as follows: ; Due to their high memory usage, existing attention-based convolutional blocks can only be applied in limited scenarios. Therefore, to explore possibilities for reducing memory usage, this embodiment... Some transformations were made. Considering... and Both are image input data The linear projection of the above equation, after transformation, yields: ; Will Set to a positive semi-definite matrix, then the attention mapping... Represented as: ; in, and This represents the parameter matrix based on a linear mapping. Represented as a four-dimensional matrix, Indexes representing different dimensions. It is a positive semi-definite matrix. The image input features represent the position (i, j). The image input features represent the location (a, b).

[0025] By Setting it to a positive semi-definite matrix allows for the determination of the key. and query It is the same, so it can save half the memory and computing overhead.

[0026] In this embodiment, as Figure 2 As shown, for a specific location Multiple regions ,like , , , respectively representing the black square portions in the corresponding areas, such as The nine black squares in the middle of the corresponding area can be used to determine this. , This corresponds to its nine black squares.

[0027] In this embodiment, a star-shaped region is used. Representing multiple regions The set, Due to the increasing costs of memory and computing... The size increases linearly. In this embodiment, multiple regions are processed by applying several extended convolutions within the convolutional block. To weigh the pros and cons.

[0028] Specifically, in a specific location Several expansion options D are defined in the convolutional block, and these expansion options D correspond to different receptive field sizes. For any expansion option, the expansion coefficient is... The corresponding area Weight based on content Integrate all regions to obtain the integrated region. , represented as: ; in, express In element.

[0029] like Figure 2 As shown, in the integration area In the diagram, the side length of the square is... D determines the size of the receptive field. The expansion coefficient for any expansion option... and its corresponding region In the middle, the corresponding receptive field size is ,at the same time The size is 9. For example... Figure 2 As shown, each d corresponds to a branch, and the colored part is... With 9 effective locations, the final size of the receptive field is: .

[0030] Based on the above process, we finally obtain: ; in, and Determined by the second channel. In this embodiment, D takes values ​​of 1, 2, and 3, representing the receptive field and... The sizes are 49 and 25, respectively. Through the above operations, the final size of the receptive field will be determined. of The computational complexity is reduced to . This refers to the computational complexity required for the convolution operation. For the convolution proposed in this embodiment, the computational complexity is related to the receptive field size. nThe relationship is of.

[0031] In this embodiment, the second channel is the Conv option branch, specifically serving as an aid to the attention branch based on the first channel. The integration region is determined within the second channel. Weights in Sum value mapping These value maps are used for .

[0032] Due to value mapping It is also a linear projection of the input, therefore this embodiment provides a matrix W. V Calculations are performed. In this embodiment, at different locations... There are different Ws V That is, W V It is content-dependent. Specifically, in the input... Apply convolution kernel W O To calculate option mapping O This mapping is done by W k and W d It consists of two parts. W k The parameter set corresponding to the convolution operation itself is represented by a matrix, W. k Representing vectors ; Therefore, for a given position The projection matrix can be obtained using the following formula. : ; Refers to W k The The element at position W k The parameter set corresponding to the convolution operation itself, W D This represents the weight set matrix.

[0033] in yes One of the predefined kernels. This means that for a given location... , exist Calculation The projection is by The content at that location determines the value mapping. The following can be calculated: ; And according to the mapping Get value mapping , Option mapping OW in D As a preset integral weight for the region in the attention branch. In this embodiment, W D Instead of using it directly, execute the maximum activation function.

[0034] ; in This represents a set of weights for different integration regions.

[0035] like Figure 3 As shown, a schematic diagram of a specific embodiment of a feature extraction method based on improved attention and dynamic convolution is presented, combined with... Figure 1 and Figure 3 Perform a single feature extraction on the image. The input is the same as the existing convolutional module. A portion of the original image is cropped using a sliding window (e.g., ...). Figure 3 The image shown depicts a square area corresponding to the dog's nose. To reduce computational load, this embodiment compresses the square receptive field into a star-shaped receptive field by occluding a portion of the receptive field, thus reducing computational complexity. Then, the inner product between different locations is calculated to obtain the correlation degree between them; this correlation degree is the attention score for each location. Figure 1 As shown, the correlation between position (i,j) and position (a,b) is... At the same time, in each location A set of weighted parameters is obtained through linear mapping. Based on this set of parameters, a predefined set of convolution parameters is weighted to generate different convolution parameters for different positions. ,in yes One of the predefined kernels. Finally, the extracted features are derived from the attention score, convolution parameters, and input. .like Figure 3 As shown, the final precise features of the dog's nose were obtained by excluding irrelevant regions and focusing on features of locations that are highly relevant to the dog's nose.

[0036] In accordance with the purpose of this embodiment, this embodiment also provides a feature extraction device based on improved attention and dynamic convolution, the feature extraction device performing the feature extraction method as given in the embodiment, the device comprising: The acquisition module is used to acquire image input data of the object whose features are to be extracted; The processing module is used to send the image input data to a first channel based on improved attention and a second channel based on dynamic convolution, respectively; The privilege extraction module is used to generate a data structure based on the integrated region through the first channel. attention mapping Weights are generated through the second channel. Sum value mapping Based on the integrated region Weight and value mapping Obtain the output data of the feature object to be extracted , represented as: Feature extraction is completed using the output data of the feature object obtained above.

[0037] This embodiment also provides an electronic device that may include a processor, a communication interface, a memory, and a communication bus, wherein the processor, communication interface, and memory communicate with each other via the communication bus. The processor can call logical instructions in the memory to execute the aforementioned feature extraction method based on improved attention and dynamic convolution.

[0038] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0039] This embodiment also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the feature extraction methods based on improved attention and dynamic convolution provided by the above methods.

[0040] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0041] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0042] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A feature extraction method based on improved attention and dynamic convolution, characterized in that, The method includes the following steps: Obtain the image input data of the object whose features are to be extracted; The image input data is fed into a first channel based on improved attention and a second channel based on dynamic convolution, respectively. generating, by the first channel, an attention map based on the integration region , generating, by the second channel, a weight and a value map , obtaining output data of the feature object to be extracted based on the integration region , the weight and the value map , represented as:​​ ； Feature extraction is completed using the image output data of the object to be extracted obtained above.

2. The feature extraction method based on improved attention and dynamic convolution according to claim 1, characterized in that, Generate an integrated region through the first channel. attention mapping The specific steps include: The attention map A is calculated based on the key K and query Q, and is represented as follows: ； After transforming the above equation, we get: ； Will Set to a positive semi-definite matrix, then the attention mapping... Represented as: ； in, and This represents the parameter matrix based on a linear mapping. Represented as a four-dimensional matrix, Indexes representing different dimensions. It is a positive semi-definite matrix. The image input features represent the position (i, j). The image input features represent the location (a, b).

3. The feature extraction method based on improved attention and dynamic convolution according to claim 1, characterized in that, Generate an integrated region through the first channel. attention mapping The steps also include: Several expansion options D are defined in the convolutional block at a specific position (i, j). These expansion options D correspond to different receptive field sizes. For any expansion option, the expansion coefficient is... The corresponding area The weights generated through the second channel Regional integration was carried out to obtain the integrated region. , is represented as: ； in, express In element.

4. The feature extraction method based on improved attention and dynamic convolution according to claim 3, characterized in that, For any expansion option, the expansion coefficient and its corresponding region In the middle, the corresponding receptive field size is .

5. The feature extraction method based on improved attention and dynamic convolution according to claim 4, characterized in that, Weights are generated through the second channel. Sum value mapping The specific steps include: Applying convolution kernels to the image input data W O To calculate option mapping O And it is expressed as follows: ； Dynamically generate a projection matrix based on the content at a specific location (i, j). And it is expressed as follows: ； Then the value mapping is calculated at the specific position (i, j). and according to the mapping Get value mapping , : ； in, express The The element at position 1 The parameter set corresponding to the convolution operation itself. W represents the weight set matrix. i This indicates a predefined kernel.

6. The feature extraction method based on improved attention and dynamic convolution according to claim 4, characterized in that, The area The size is 9.

7. The feature extraction method based on improved attention and dynamic convolution according to claim 6, characterized in that, The integrated area Including star-shaped areas The star-shaped region Represented as: ； Among them, the star-shaped region The size is represented as: .

8. A feature extraction device based on improved attention and dynamic convolution, characterized in that, The feature extraction apparatus performs the feature extraction method as described in any one of claims 1-7, the apparatus comprising: The acquisition module is used to acquire image input data of the object whose features are to be extracted; The processing module is used to send the image input data to a first channel based on improved attention and a second channel based on dynamic convolution, respectively; The privilege extraction module is used to generate a data structure based on the integrated region through the first channel. attention mapping Weights are generated through the second channel. Sum value mapping Based on the integrated region Weight and value mapping Obtain the output data of the feature object to be extracted , is represented as: ； Feature extraction is completed using the output data of the object to be extracted obtained above.

9. An electronic device, characterized in that, The method includes a memory and a processor, characterized in that the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the feature extraction method based on improved attention and dynamic convolution as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the feature extraction method based on improved attention and dynamic convolution as described in any one of claims 1-7.

Images