Adaptive global pooling attention method and device, electronic equipment and storage medium

Abstract

The application discloses a kind of self-adapting global pooling attention method, device, electronic equipment and storage medium, method includes: by parallel global feature extraction module, global average pooling and global maximum pooling are carried out to input feature map, to extract the global feature of input feature map;Global feature is sent into adaptive learning module, and the global pooling feature optimized after learning is obtained by adaptive learning module;Optimized global pooling feature is used to guide input feature map to obtain optimized output feature.The self-adapting global pooling attention method of the application adopts the mode that global maximum pooling and global average pooling are combined, and introduces adaptive learning mechanism, and dynamically adjusts the weight size that global maximum feature and global average feature are occupied when guiding input.At the same time when considering the overall feature distribution of image, important local feature is emphasized, and more robust feature representation can be provided.Can focus on the key feature that has important influence on image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a self-adaptive global pooling attention method and device, electronic equipment and storage medium. BACKGROUND

[0002] Target detection is one of the popular research directions in the field of computer vision, and its main task is to locate all the targets of interest in the image and determine their types and positions. It is widely used in pedestrian detection, automatic driving, video monitoring and other fields. However, target detection still faces many challenges, including target occlusion, background interference, etc. Researchers have been actively responding to these challenges and trying various methods to improve the performance of target detection algorithms.

[0003] The introduction of attention mechanism can solve the challenges faced by target detection to some extent. It can help the model focus on important areas or features when processing images, thereby improving the accuracy of target detection. Common attention mechanisms can be divided into channel attention, spatial attention, squeeze-and-excitation (SE) attention. SE is a relatively classic channel attention, and its core idea is to learn the weights of different channels through compression and excitation operations and highlight significant features. However, the shortcomings of SE attention are also obvious, which ignores the importance of spatial information. Convolutional block attention module (CBAM) integrates channel attention and spatial attention into one module, which improves the performance. Some researches try to simplify the complexity of attention model, for example, efficient channel attention (ECA) simplifies the calculation process of channel attention weight in SE through one-dimensional convolution. Shuffle attention (SA) uses branch structure to divide the channel dimension into multiple sub-features, and then effectively integrates complementary channel and spatial attention for each sub-feature through shuffle unit. Coordinate attention (CA) embeds direction-related position information into channel attention information, which reduces the parameters while improving the detection accuracy.

[0004] Although the detection accuracy can be improved by integrating these attention methods into the target detection algorithm, these attention methods mostly use a single global pooling method such as global max pooling (GMP) or global average pooling (GAP), which cannot capture more representative global features and is difficult to significantly improve the performance of the target detection network. Generally, attention mostly uses a simple global pooling method, which cannot capture more representative global features. If only global average pooling is used to extract global features, some edge features will be blurred; if only global max pooling is used to extract global features, the overall features will be ignored; if only a simple weighting of global max pooling and global average pooling is used, although the edge and overall features are considered comprehensively, the weight size of the global max feature and the global average feature in guiding the input cannot be adjusted adaptively. SUMMARY

[0005] The present application provides an adaptive global pooling attention method, device, electronic equipment and storage medium to solve the above technical problems in the prior art.

[0006] The technical solution adopted by the present application is to provide an adaptive global pooling attention method, comprising:

[0007] The input feature map is subjected to global average pooling and global max pooling by the parallel global feature extraction module to extract the global features of the input feature map;

[0008] The global features are sent to the adaptive learning module, and the adaptive learning module learns the optimized global pooling features;

[0009] The input feature map is guided by the optimized global pooling features to obtain optimized output features.

[0010] Further, the method of extracting global features of the input feature map by the parallel global feature extraction module comprises:

[0011] The global average pooling and global max pooling are performed on each channel of the input feature map to extract the average features and the maximum features of the input, respectively;

[0012] The one-dimensional convolution is used to cross-channel interaction process the average features and the maximum features, respectively, to capture the cross-channel correlation, integrate the information of different channels, and provide more comprehensive and consistent feature representation; the one-dimensional convolution cross-channel processing obtains the overall features and the significant features; the mathematical model expression of the parallel global feature extraction module is shown as (1)-(4):

[0013]

[0014] z m =F max (X)=argmax(C(i,j)) (2)

[0015] OF = conv k＝5 (z a (3)

[0016] PF = conv k＝5 (z m (4)

[0017] In the formula z a z represents the average characteristic. m Let i = 1, 2, 3, ..., H, j = 1, 2, 3, ..., W, conv denote one-dimensional convolution, k = 5 indicates that the convolution kernel size is 5, OF denotes the overall feature, and PF denotes the salient feature; C, H, and W are the number of channels, height, and width of the input feature map X, respectively, and C(i,j) represents the feature value of the input feature map at position (i,j).

[0018] Furthermore, the method for feeding the global features into the adaptive learning module, and for the adaptive learning module to learn the optimized global pooling features, includes:

[0019] The overall features and salient features obtained by the parallel global feature extraction module are fed into the adaptive learning module, which learns the optimized global pooling features. The adaptive learning module includes: an enhanced feature learning module, a differential feature learning module, and a fusion feature learning module.

[0020] The expression for the enhanced feature learning module is shown in (5):

[0021] EF=α×OF+(1-α)×PF (5)

[0022] The enhanced feature EF is obtained by automatically adjusting the enhanced feature relationship between the overall feature OF and the salient feature PF using the enhanced feature operator α; the value of α is between 0 and 1 and is learnable, and its value is updated in each backpropagation process.

[0023] The expression for the differential feature learning module is shown in (6):

[0024] DF=β×AF-(1-β)×MF (6)

[0025] The difference feature DF is obtained by automatically adjusting the difference feature relationship between the overall feature OF and the salient feature PF using a difference feature operator β; the value range of β is between 0 and 1, and β is learnable, and the value of β is updated in each back propagation process;

[0026] The expression of the fusion feature learning module is shown in (7):

[0027] FF = γ × DF + (1 - γ) × EF (7)

[0028] The fusion feature FF is obtained by automatically adjusting the fusion feature relationship between the enhanced feature EF and the difference feature DF using a fusion feature operator γ; the value range of γ is between 0 and 1, and γ is learnable, and the value of γ is also updated in each back propagation process.

[0029] In a second aspect, an adaptive global pooling attention device is provided, comprising:

[0030] A parallel global feature extraction module is configured to perform global average pooling and global maximum pooling on an input feature map to extract global features of the input feature map;

[0031] An adaptive learning module is configured to send the global features to the adaptive learning module, and the adaptive learning module learns optimized global pooling features;

[0032] An output feature module is configured to guide the input feature map according to the optimized global pooling features to obtain optimized output features.

[0033] In a third aspect, an electronic device is provided, comprising:

[0034] One or more processors;

[0035] A storage device configured to store one or more programs,

[0036] When the one or more programs are executed by the one or more processors, the one or more processors implement the adaptive global pooling attention method according to the first aspect.

[0037] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the program is executed by a processor to implement the adaptive global pooling attention method according to the first aspect.

[0038] The beneficial effects of the present application are: the adaptive global pooling attention method of the present application adopts a combination of global maximum pooling and global average pooling, and introduces an adaptive learning mechanism to dynamically adjust the weight size of global maximum features and global average features when guiding input. The present application emphasizes important local features while considering the overall feature distribution of the image, and can provide more robust feature representation. It can focus attention on key features that have a significant impact on the image, highlight the differences in details in the image, and provide more rich and detailed feature representation. The fused features combine enhanced features and difference features, complement each other, and provide more comprehensive and rich feature information. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 The overall structure schematic diagram of the adaptive global pooling attention disclosed by the present application is shown in the figure.

[0040] Figure 2 The schematic diagram of the parallel global feature extraction module disclosed by the present application is shown in the figure.

[0041] Figure 3 The schematic diagram of the adaptive learning module disclosed by the present application is shown in the figure.

[0042] Figure 4 The structure schematic diagram of the electronic device disclosed by the present application is shown in the figure. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings, but the embodiments of the present application are not limited thereto.

[0044] Example 1

[0045] The present embodiment discloses an adaptive global pooling attention (AGPA) method, comprising the following steps:

[0046] Step 1: Perform global average pooling and global maximum pooling on the input feature map through the parallel global feature extraction module to extract the global features of the input feature map.

[0047] Specifically, Figure 2The parallel global feature extraction module is shown in the schematic diagram. C, H, and W in the input feature map X are the channel number, height, and width, respectively. Each channel of the input feature map is an independent HxW matrix. Different channels are different edges, textures, shapes, or other visual features. First, the global average pooling (GAP) and global max pooling (GMP) are performed on each channel of the input feature map X to extract the average feature and the maximum feature of the input, respectively. Then, a one-dimensional convolution with a kernel size of 5 is used to cross-channel interaction process the average feature and the maximum feature, respectively, to capture the cross-channel correlation, integrate the information of different channels, and provide more comprehensive and consistent feature representation. The one-dimensional convolution cross-channel processing obtains the overall feature (OF) and the prominent feature (PF). The mathematical model expressions of the parallel global feature extraction module are shown in equations (1)-(4):

[0048]

[0049] z m =F max (X)=argmax(C(i,j)) (2)

[0050] OF=conv k＝5 (z a ) (3)

[0051] PF=conv k＝5 (z m ) (4)

[0052] In the equations, z a represents the average feature, z m represents the maximum feature, i = 1, 2, 3,..., H, j = 1, 2, 3,..., W, conv represents one-dimensional convolution, k = 5 represents the kernel size of 5, OF represents the overall feature, PF represents the prominent feature, C, H, and W are the channel number, height, and width of the input feature map X, respectively, and C(i,j) represents the feature value of the input feature map at the (i,j) position.

[0053] Step 2: The global feature is sent to the adaptive learning module, and the adaptive learning module learns the optimized global pooling feature.

[0054] Specifically, as Figure 3As shown in FIG. 3, the schematic diagram of the adaptive learning module, the overall feature and the salient feature obtained by the parallel global feature extraction module are sent into the adaptive learning module, and the optimized global pooling feature is learned by the adaptive learning module. The adaptive learning module is composed of three parts: an enhanced feature learning module, a differential feature learning module, and a fused feature learning module.

[0055] (1) Enhanced feature learning module. The expression of the enhanced feature learning module is shown in (5). The enhanced feature operator a is used to automatically adjust the enhanced feature relationship between the overall feature OF and the salient feature PF to obtain the enhanced feature EF. The value of a ranges from 0 to 1 and is learnable, and its value is updated in each backpropagation process. The enhanced feature combines the advantages of the overall feature and the salient feature, emphasizes important local features while considering the overall feature distribution of the image, and can provide more robust feature representation.

[0056] EF = a x OF + (1 - a) x PF (5)

[0057] (2) Differential feature learning module. The expression of the differential feature learning module is shown in (6). The differential feature operator b is used to automatically adjust the differential feature relationship between the overall feature OF and the salient feature PF to obtain the differential feature DF. The value of b ranges from 0 to 1 and is learnable, and its value is updated in each backpropagation process. The differential feature can capture feature values with large differences between the overall feature and the salient feature, thereby focusing attention on key features that have important influences on the image, highlighting the detail differences in the image, and providing more rich and detailed feature representation.

[0058] DF = b x AF - (1 - b) x MF (6)

[0059] (3) Fused feature learning module. The expression of the fused feature learning module is shown in (7), and the fused feature operator g is used to automatically adjust the fused feature relationship between the enhanced feature EF and the differential feature DF to obtain the fused feature FF. The value of g ranges from 0 to 1 and is learnable, and its value is also updated in each backpropagation process. The fused feature combines the enhanced feature and the differential feature, and provides more comprehensive and rich feature information. The feature after fusion is the optimized global pooling feature.

[0060] FF = g x DF + (1 - g) x EF (7)

[0061] Step 3: The input feature map is guided by the optimized global pooling feature to obtain the optimized output feature.

[0062] Specifically, as Figure 1 The overall structure of the adaptive global pooling attention is shown in the schematic diagram. The optimized global pooling features are used to guide the input feature map to obtain the optimized output features. The formula of the overall adaptive global pooling attention module is shown in equation (8):

[0063] X' = FF x X (8)

[0064] In the above formula, a, b, and g are all learnable parameters, and the initial values are subject to a random standard normal distribution f(x), with a value range of 0-1, where x is a random number. The formula is shown in equation (9):

[0065]

[0066] The updated value of the model during each backpropagation is also limited to 0-1 using the Sigmoid function.

[0067] Experiments were conducted on the public datasets VOC2007 and COCO2017. The disclosed datasets can verify whether the proposed attention module improves the performance of the target detection network. In this experiment, the PyTorch framework was used. During training, the standard SGD optimizer was used with a momentum parameter of 0.92, an optimizer weight decay of 2e-4, an initial learning rate of 0.01, a final learning rate of 0.2, a batch size of 32, an input image size of 640x640, an iteration number of 3, an initial momentum parameter of 0.8, and an initial bias of 0.1 to warm up the learning rate.

[0068] Various mainstream attention methods were added to the benchmark models of YOLOv5s and YOLOv7-tiny, and compared with the adaptive global pooling attention AGPA. In the table, ↑ indicates the value of the index improvement compared with the benchmark model. The experimental results are shown in Tables 1 and 2. The method (AGPA) of the present application outperforms the current mainstream attention methods on the VOC2007 dataset and the COCO2017 dataset. The adaptive global pooling attention AGPA has a significant improvement in each mAP index when the parameters and GFLOPs are small.

[0069] Table 1 Comparison of different attention methods on VOC2007

[0070]

[0071]

[0072] Table 2 Comparison of different attention methods on COCO2017

[0073]

[0074] Embodiment 2

[0075] The embodiment discloses an adaptive global pooling attention device, namely an adaptive global pooling attention (AGPA) module, which is composed of three sub-modules: a parallel global feature extraction module, an adaptive learning module and an output feature module.

[0076] The parallel global feature extraction module is used for performing global average pooling and global maximum pooling on the input feature map to extract global features of the input feature map.

[0077] The adaptive learning module is used for sending the global features into the adaptive learning module, and learning optimized global pooling features by the adaptive learning module.

[0078] The output feature module is used for guiding the input feature map according to the optimized global pooling features to obtain optimized output features.

[0079] The adaptive global pooling attention device can execute the adaptive global pooling attention method provided in Embodiment 1, and has corresponding functional modules and beneficial effects of executing the adaptive global pooling attention method.

[0080] Embodiment 3

[0081] Figure 4 A structural schematic diagram of an electronic device is provided for the embodiment. Figure 4 A block diagram of an exemplary electronic device 40 suitable for implementing an embodiment of the present application is shown. Figure 4 The electronic device 40 shown is merely an example and should not bring any limitation to the function and use range of the embodiment of the present application.

[0082] As shown in Figure 4 The electronic device 40 is in the form of a general computing device. The components of the electronic device 40 can include, but are not limited to, one or more processors or processing units 401, system memory 402, and a bus 403 connecting different system components, including the system memory 402 and the processing unit 401.

[0083] Bus 403 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration bus, a processor or local bus using any of a variety of bus architectures. By way of example, these architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0084] Electronic device 40 typically includes a variety of computer system readable media. These media can be any available media that is accessible by electronic device 40 and includes both volatile and nonvolatile media, removable and non-removable media.

[0085] The system memory 402 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 404 and / or cache memory 405. Electronic device 40 can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 406 can be provided for reading from and writing to non-removable, non-volatile magnetic media (e.g., a "hard drive"). Figure 4 not shown, a magnetic hard disk drive for reading from and writing to non-removable, non-volatile magnetic media (e.g., a "hard drive"). Although not specifically shown, such Figure 4 In alternative embodiments, a magnetic hard disk drive, a solid state drive (SSD) which is a non- volatile storage medium, a floppy disk drive for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading from or writing to a removable, non- volatile optical disk (e.g., a CD-ROM, DVD-ROM, and other optical media) can be provided. In such instances, each drive can be connected to the system bus 403 by one or more data media interfaces. The memory 402 can include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the application.

[0086] Program / utility 408, having a set (at least one) of program modules 407, can be stored in memory 402 by way of example, such as an operating system, one or more application programs, other program modules, and program data, each of which implements aspects of the application, as well as possibly a graphical user interface. Generally, program modules include routines, programs, components, data structures, objects, and the like, which perform particular tasks and / or implement particular abstract data types.

[0087] The electronic device 40 can also communicate with one or more external devices 409, such as a keyboard or a pointing device, a display 410, etc.; other devices Figure 4 Other hardware and / or software modules that can be used in conjunction with the electronic device 40, such as microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc. are not shown in FIG. 4, but can be readily incorporated.

[0088] The processing unit 401 executes various function applications and data processing by running programs stored in the system memory 402, such as implementing the adaptive global pooling attention method provided by Embodiment 1.

[0089] Embodiment 4:

[0090] The embodiment provides a storage medium containing computer executable instructions, which when executed by a computer processor, are used to perform an adaptive global pooling attention method.

[0091] The computer storage medium of the embodiment of the application can adopt any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium may, for example, be but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples (non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus.

[0092] A computer readable signal medium can include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal can take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium can be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate or transport program code. Program code embodied on a computer readable signal medium can be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, R.F, etc., or any suitable combination of the foregoing.

[0093] Computer readable program code embodied on a computer readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, R.F, etc., or any suitable combination of the foregoing.

[0094] Computer program code for carrying out operations of embodiments of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In an embodiment of the application, electronic mail (email) can be used to perform some or all of the communication steps between the user and the remote computer.

[0095] The above embodiments are only used to illustrate the technical solutions of the present application, not limit the scope of the present application; although the present application is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: still can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An adaptive global pooling attention method, characterized in that, include: Step 1: The input is an image. The parallel global feature extraction module performs global average pooling and global max pooling on each channel of the input feature map to extract the average feature and maximum feature of the input respectively. The mathematical model expressions of the parallel global feature extraction module are shown in (1)-(4): (1)； (2)； (3)； (4)； In the formula, z a Indicates average characteristics; z m The largest feature is represented by i = 1, 2, 3, ..., H; j = 1, 2, 3, ..., W; conv represents one-dimensional convolution; k = 5 indicates that the kernel size is 5; H and W are the height and width of the input feature map X, respectively; C(i,j) represents the feature value of the input feature map at position (i,j); OF represents the overall feature, and PF represents the salient feature. One-dimensional convolution is used to perform cross-channel interactive processing on the average feature and the maximum feature respectively, capturing the correlation between channels and integrating the information of different channels to provide a more comprehensive and consistent feature representation; the one-dimensional convolution cross-channel processing yields the overall feature and the salient feature. Step 2: The overall features and the salient features are fed into the adaptive learning module, which learns the optimized global pooling features. The adaptive learning module includes: an enhanced feature learning module, a differential feature learning module, and a fusion feature learning module; The expression for the enhanced feature learning module is shown in (5): (5)； The enhanced feature EF is obtained by automatically adjusting the enhanced feature relationship between the overall feature OF and the salient feature PF using the enhanced feature operator α; the value of α is between 0 and 1 and is learnable, and its value is updated in each backpropagation process. The expression for the differential feature learning module is shown in (6): (6)； The differential feature DF is obtained by automatically adjusting the differential feature relationship between the global feature OF and the salient feature PF using the differential feature operator β; the value of β is between 0 and 1 and is learnable, and its value is updated in each backpropagation process. The expression for the fusion feature learning module is shown in (7): (7)； The fusion feature FF is obtained by automatically adjusting the fusion feature relationship between the enhanced feature EF and the differential feature DF using the fusion feature operator γ; the value of γ is between 0 and 1 and is learnable, and its value is updated in each backpropagation process. Step 3: Use the optimized global pooling features to guide the input feature map to obtain the optimized output features.

2. An adaptive global pooling attention device, characterized in that, To implement the adaptive global pooling attention method as described in claim 1, the method includes: A parallel global feature extraction module is used to perform global average pooling and global max pooling on the input feature map to extract global features from the input feature map; An adaptive learning module is used to feed the global features into the adaptive learning module, and the adaptive learning module learns the optimized global pooling features. The output feature module is used to guide the input feature map based on the optimized global pooling features to obtain the optimized output features.

3. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the adaptive global pooling attention method as described in claim 1.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the adaptive global pooling attention method as described in claim 1.

Images