Feature enhancement method and device based on attention mechanism and image processing method

Abstract

The application discloses a feature strengthening method and device based on an attention mechanism and an image processing method, and relates to the technical field of image processing.The feature strengthening method comprises the following steps: obtaining an initial feature matrix to be strengthened of an image; performing feature strengthening processing on the initial feature matrix to obtain a first feature matrix subjected to feature strengthening attention weighting; performing feature space correlation extraction and processing on the initial feature matrix to obtain a second feature matrix subjected to space correlation attention weighting; performing feature channel correlation extraction and processing on the initial feature matrix to obtain a third feature matrix subjected to channel correlation attention weighting; and weighting the first feature matrix, the second feature matrix and the third feature matrix to obtain a strengthened feature matrix.The application can be applied to improving the performance of a lightweight convolutional neural network, improving the accuracy of image classification, target detection, behavior recognition and instance segmentation and the like under the premise of ensuring the lightweight of the parameters of the lightweight convolutional neural network, and meeting the basic requirements in application.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a feature strengthening method and device based on an attention mechanism and an image processing method. BACKGROUND

[0002] With the advent of the network era and the acceleration of information transmission speed, images and videos gradually show unique advantages that text does not have, and are more and more welcomed and used by people. At the same time, artificial intelligence develops rapidly, and the computing power of computers is also increasing. Under such a background, the understanding of images and videos through computer programs also brings great convenience to people's life. Most of the existing excellent image processing algorithms use deep neural networks, especially convolutional neural networks. According to the size of the parameter quantity, the convolutional neural network can be divided into a neural network of ordinary order of magnitude and a light-weight neural network.

[0003] Common general-purpose convolutional neural networks of ordinary order of magnitude include AlexNet, GoogleNet, ResNet and DenceNet, etc. These networks have the advantages of simple structure and strong universality, and the general parameter quantity is in the order of tens of megabytes. In the development process of neural networks, on the one hand, it is the research direction of performance improvement. Recently, the popular self-attention network Transformer and various multi-flow combined convolutional models have a parameter quantity of hundreds of megabytes, and the accuracy has also increased significantly. The other research direction is the light-weight network developed for mobile devices, which aims to improve the performance as much as possible while ensuring that the parameter quantity is small enough. This part of the network generally has a parameter quantity of less than ten megabytes, and can be applied and carried on mobile devices for use. The current most advanced light-weight convolutional network includes HBONet, ShuffleNetV2, MobileNetV3 and EfficientNet, etc.

[0004] An effective way to improve the performance of modern deep neural networks is attention mechanism, which has been proven to be effective in various computer vision tasks. Attention mechanism can be regarded as a dynamic selection process that is achieved by weighting features according to their importance, and its essence is to guide the model to focus on some regions of the image that can provide effective features. Since deep neural networks lack interpretability, and the extraction of image features is automatically completed, attention mechanism also needs to have adaptive characteristics, which makes the research of attention mechanism a complex research. There are many existing researches on attention mechanism, whether it is channel-based attention, spatial-based attention, or the recently popular self-attention, all of which can effectively improve the performance of deep neural networks, but most of these methods will bring an unbearable parameter cost to lightweight networks. Therefore, it is valuable to carry out research on image feature enhancement based on attention mechanism for lightweight networks.

[0005] Research on attention mechanism image feature enhancement applied to lightweight deep networks is a fundamental work. By using attention mechanism to improve network performance, feature enhancement processing is performed on the feature matrix in the convolution process of lightweight deep networks, so that lightweight networks can alleviate the irreconcilable contradiction between parameter quantity and performance, and have higher freedom at the trade-off point. This can promote the research of lightweight networks to have greater breakthroughs in structure and method, and can also inspire researchers who study attention mechanism in ordinary level deep networks in research ideas. The basic task of using lightweight networks on images is image classification task, and researchers also usually apply these networks to downstream tasks such as object detection, behavior recognition, instance segmentation, etc. as backbone networks. Therefore, the attention mechanism method based on lightweight networks can be valuable in various tasks.

[0006] In fact, the feature enhancement method of attention mechanism designed around lightweight networks is not just an academic research, but also has an important position in real life and application. Due to parameter restrictions, lightweight networks have to be used on various mobile devices in the past, which has largely lost the advantages of deep networks in performance, seriously affecting the user experience. By constructing a suitable attention mechanism, the performance improvement of lightweight networks can effectively optimize its application range and user experience.

[0007] The most widely used method in recent lightweight neural networks is the squeeze-and-excitation (SE) channel attention method. There are also some spatial domain-based attention methods such as BAM and the like. However, these methods only consider the relationship of image features in the encoding channel or local space, which severely limits the improvement of network performance. At the same time, these methods have already put a certain pressure on the lightweight network in terms of parameters, and blindly combining these methods to enhance image features from multiple angles will cause the parameter quantity of the network to face a multiple increase. Therefore, it is meaningful to develop an image feature enhancement method combining spatial domain and channel domain attention mechanisms around lightweight neural networks. SUMMARY

[0008] In view of the deficiencies in the prior art, the present application provides a feature enhancement method based on an attention mechanism and an image processing method, which can be applied to improve the performance of a lightweight convolutional neural network, improve the accuracy of image classification, target detection, behavior recognition and instance segmentation under the premise of ensuring the lightweight of the parameters of the lightweight convolutional neural network, and meet the basic needs in applications.

[0009] To achieve the above-mentioned purpose, the present application provides a feature enhancement method based on an attention mechanism, comprising the following steps:

[0010] Step 1, obtaining an initial feature matrix of an image to be enhanced;

[0011] Step 2, performing feature enhancement processing on the initial feature matrix to obtain a first feature matrix weighted by feature enhancement attention;

[0012] Step 3, extracting and processing the feature spatial correlation of the initial feature matrix to obtain a second feature matrix weighted by spatial correlation attention;

[0013] Step 4, extracting and processing the feature channel correlation of the initial feature matrix to obtain a third feature matrix weighted by channel correlation attention;

[0014] Step 5, weighting the first feature matrix, the second feature matrix and the third feature matrix to obtain an enhanced feature matrix.

[0015] In one embodiment, step 2 specifically comprises:

[0016] Step 2.1, extracting spatial domain information: performing one-dimensional average pooling on the channel domain of the initial feature matrix, so that the initial feature matrix with a size of C*H*W is converted into a first spatial domain matrix with a size of 1*H*W;

[0017] Step 2.2, amplifying the representation difference ratio of the features: performing an element-wise squaring operation on the first spatial domain matrix with a size of 1*H*W to amplify the representation difference of the spatial domain features, to obtain a second spatial domain matrix;

[0018] Step 2.3, weight acquisition: processing the second spatial domain matrix by using a Sigmoid activation function to obtain a first attention weight matrix with a size of 1*H*W, copying the first attention weight matrix to obtain a second attention weight matrix with a size of C*H*W;

[0019] Step 2.4, weighted calculation: element-wise multiplying the second attention weight matrix and the initial feature matrix to obtain a first feature matrix with a size of C*H*W and a feature-enhanced attention weighting.

[0020] In one of the embodiments, step 3 specifically comprises:

[0021] Step 3.1, extracting spatial domain information: one-dimensional average pooling is performed on the channel domain of the initial feature matrix, so that the initial feature matrix with a size of C*H*W is converted into a third spatial domain matrix with a size of 1*H*W;

[0022] Step 3.2, matrix vectorization: vectorization is performed on the first spatial domain matrix with a size of 1*H*W, so that the third spatial domain matrix with a size of 1*H*W is converted into a first feature vector with a size of 1*(H*W);

[0023] Step 3.3, acquisition and processing of feature space correlation matrix: matrix multiplication is performed on the transposed first feature vector and itself to obtain a feature space correlation matrix with a size of (H*W)*(H*W), summing operation is performed along any dimension of the feature space correlation matrix to obtain a spatial correlation vector with a size of 1*(H*W), and then the spatial correlation vector is restored to a feature matrix with a size of 1*H*W, that is, a fourth spatial domain matrix containing global information is obtained;

[0024] Step 3.4, weight acquisition: processing the fourth spatial domain matrix by using a Sigmoid activation function to obtain a third attention weight matrix with a size of 1*H*W, copying the third attention weight matrix to obtain a fourth attention weight matrix with a size of C*H*W;

[0025] Step 3.5, weighted calculation: element-wise multiplying the fourth attention weight matrix and the initial feature matrix to obtain a second feature matrix with a size of C*H*W and a spatial correlation attention weighting.

[0026] In one of the embodiments, step 4 specifically comprises:

[0027] Step 4.1, extracting channel domain information: two-dimensional average pooling is performed on the spatial domain of the initial feature matrix, and a dimension with a size of 1 is removed, so that the initial feature matrix with a size of C*H*W is converted into a second feature vector with a size of C*1;

[0028] Step 4.2, acquisition and processing of channel correlation information: determine the channel correlation attention width C according to the number of feature channels, and adaptively confirm the parameter k according to the channel width C, perform one-dimensional convolution operation on the second feature vector using a convolution kernel with a size of k, and obtain a third feature vector with a size of C*1 and containing channel interaction information;

[0029] Step 4.3, weight acquisition: process the third feature vector using a Sigmoid activation function to obtain a fifth attention weight matrix with a size of C*1, and copy the fifth attention weight matrix to obtain a sixth attention weight matrix with a size of C*H*W;

[0030] Step 4.4, weighted calculation: multiply the sixth attention weight matrix and the initial feature matrix element by element to obtain a third feature matrix with a size of C*H*W and channel correlation attention weighted.

[0031] In one embodiment, in step 4.2, the parameter k is adaptively confirmed according to the channel width C, which is specifically:

[0032]

[0033] In the formula, |·| odd Indicates the operation of taking the nearest odd number from the calculation result.

[0034] In one embodiment, in step 5, the enhanced feature matrix is specifically:

[0035] X=a·X1+b·X2+(1-a-b)·X3

[0036] In the formula, X is the enhanced feature matrix after feature enhancement processing, X1 is the first feature matrix, X2 is the second feature matrix, X3 is the third feature matrix, and a and b are hyperparameters.

[0037] To achieve the above purpose, the application provides a feature enhancement device based on an attention mechanism, which uses the above method to enhance image features. The feature enhancement device comprises:

[0038] A feature matrix acquisition unit is configured to acquire an initial feature matrix of an image to be enhanced.

[0039] A feature enhancement unit is configured to perform feature enhancement processing on the initial feature matrix to obtain a first feature matrix weighted by feature enhancement attention.

[0040] A spatial correlation enhancement unit is configured to extract and process the feature spatial correlation of the initial feature matrix to obtain a second feature matrix weighted by spatial correlation attention.

[0041] The channel correlation reinforcement unit extracts and processes the feature channel correlation of the initial feature matrix to obtain a third feature matrix subjected to channel correlation attention weighting.

[0042] The feature output unit is configured to weight the first feature matrix, the second feature matrix and the third feature matrix to obtain a strengthened feature matrix and output the strengthened feature matrix.

[0043] To achieve the above object, the present application provides an image processing method, which is image classification or target detection or behavior recognition or instance segmentation, and adopts a lightweight convolutional neural network to process the image, and the process is as follows:

[0044] Step one: input a target image into the lightweight convolutional neural network;

[0045] Step two: based on the network structure of the lightweight convolutional neural network, use a convolution block with an N*N convolution kernel to perform convolution processing on the image feature matrix of the target image;

[0046] Step three: in the repeated convolution process of the network, add the feature strengthening device mentioned above in each convolution block, and then use attention to guide feature construction in the process of extracting the feature matrix in the convolution layer;

[0047] Step four: based on the network structure of the lightweight convolutional neural network, use 1*1 convolution operation and pooling operation to process the feature matrix output by the last convolution block and the feature strengthening device in sequence, to improve the semantic expressiveness of the features;

[0048] Step five: perform the last processing on the integrated feature matrix, so as to calculate and convert it to the format required by the task, and output the result of image processing.

[0049] The feature strengthening method, device and image processing method based on the attention mechanism provided by the present application can adaptively extract and weight the image features, improve the sensitivity of the network to the key positions and key targets of the image, enrich the distinguishability of the output features of the network, and can be applied to improve the performance of the lightweight convolutional neural network, improve the accuracy of image processing tasks such as image classification, target detection, behavior recognition and instance segmentation under the premise of ensuring the lightweight of the parameters of the lightweight convolutional neural network, and meet the basic requirements in applications. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to the structures shown in these drawings without any creative effort.

[0051] Figure 1 Flow chart for feature enhancement method in embodiment 1 of the present application;

[0052] Figure 2 Frame chart for feature enhancement device in embodiment 2 of the present application;

[0053] Figure 3 Flow chart for image processing in embodiment 3 of the present application.

[0054] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0056] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present application are only used to explain the relative positional relationship, movement condition, etc. between components in a certain posture (as shown in the drawings), and if the certain posture changes, the directional indications also change accordingly.

[0057] In addition, the description such as "first", "second" and the like in the present application is only for the purpose of description, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, such as two, three, etc., unless otherwise specifically limited.

[0058] In the present application, unless otherwise specifically defined and limited, the terms "connection", "fixation" and the like should be understood in a broad sense, for example, "fixation" can be fixed connection, or detachable connection, or integral; can be mechanical connection, or electrical connection, or physical connection or wireless communication connection; can be directly connected, or indirectly connected through an intermediate medium, or the internal communication of two elements or the interaction relationship between two elements, unless otherwise specifically limited. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0059] In addition, the technical solutions among various embodiments of the present application can be combined with each other, but the combination of the technical solutions should be considered not to exist and not within the protection scope of the present application on the basis that the combination of the technical solutions can be realized by the ordinary skilled in the art, and the combination of the technical solutions cannot be realized or contradicts each other.

[0060] Embodiment 1

[0061] The embodiment discloses a feature strengthening method based on an attention mechanism, mainly applied to image feature strengthening of a lightweight convolutional neural network in image processing tasks such as image classification, target detection, behavior recognition and instance segmentation, and improves the accuracy of the tasks such as image classification, target detection, behavior recognition and instance segmentation under the premise of ensuring the lightweight of the network parameters of the lightweight convolutional neural network. Figure 1 The feature strengthening method comprises the following steps:

[0062] Step 1, obtaining an initial feature matrix to be strengthened of an image. In the specific implementation process, the initial feature matrix can be a convolutional feature obtained through a convolutional layer of the lightweight convolutional neural network.

[0063] Step 2, performing feature strengthening processing on the initial feature matrix, obtaining an attention weight in a spatial domain, and obtaining a first feature matrix after feature strengthening attention weighting. Since the lightweight convolutional neural network is far inferior to an ordinary level deep neural network in terms of the depth of the network layer and the parameter complexity of each feature extraction module, a lot of information useless for image classification is easily extracted as features in the process of extracting image features. Therefore, simple feature strengthening processing can make the network more easily distinguish the information effective or ineffective for classification in the image. Therefore, in the embodiment, the specific implementation process of the feature strengthening processing is as follows:

[0064] Step 2.1, extracting spatial domain information: performing one-dimensional average pooling on the channel domain of the initial feature matrix, removing the channel domain information, and converting the initial feature matrix with a size of C*H*W into a first spatial domain matrix with a size of 1*H*W, so as to facilitate subsequent processing of the information in the spatial domain; wherein, the average of the channel information is beneficial to analyze the tendency of the network in the image information extraction from a comprehensive point of view, and can save computing resources and improve network operation efficiency in subsequent spatial domain feature strengthening;

[0065] Step 2.2, amplifying the representation difference ratio of the feature: performing an element-by-element squaring operation on the first spatial domain matrix with a size of 1*H*W to amplify the representation difference of the spatial domain feature, and obtaining a second spatial domain matrix; wherein, the essence of the image feature is the result of information screening in the process of image content extraction, and the amplification of the feature representation difference can effectively eliminate the influence of invalid information on the network performance in view of the feature information mixing caused by too few network layers of the lightweight network.

[0066] Step 2.3, weight acquisition: the second spatial domain matrix is processed by using a Sigmoid activation function to obtain a first attention weight matrix with a size of 1*H*W, the first attention weight matrix is copied to obtain a second attention weight matrix with a size of C*H*W;

[0067] Step 2.4, weighted calculation: the second attention weight matrix is multiplied with the initial feature matrix element by element to obtain a first feature matrix with a size of C*H*W and a feature-enhanced attention weighting.

[0068] Step 3, the initial feature matrix is extracted and processed for feature space correlation, an attention weight of a spatial domain is obtained, and a second feature matrix after spatial correlation attention weighting is obtained. Due to the local characteristics of the convolution operation of the convolutional neural network, it is difficult to establish the spatial relationship of the features at the global level of the image, and the global information will be continuously lost while the network depth increases, and this part of information is valuable for image classification. Therefore, the capture and processing module of the spatial correlation is added after each convolution block to supplement the image information lost by the convolution features. In this embodiment, the specific implementation process of the extraction and processing of the feature space correlation is as follows:

[0069] Step 3.1, extract spatial domain information: one-dimensional average pooling is performed on the channel domain of the initial feature matrix to remove the channel domain information, so that the initial feature matrix with a size of C*H*W is converted into a third spatial domain matrix with a size of 1*H*W;

[0070] Step 3.2, matrix vectorization: the first spatial domain matrix with a size of 1*H*W is subjected to vectorization operation, so that the third spatial domain matrix with a size of 1*H*W is converted into a first feature vector with a size of 1*(H*W); this can facilitate the subsequent operation of extracting the feature space correlation;

[0071] Step 3.3, acquisition and processing of feature space correlation matrix: the transposed first feature vector is multiplied with itself to obtain a feature space correlation matrix with a size of (H*W)*(H*W), and a sum operation is performed along any dimension of the feature space correlation matrix (for example, sum along the first dimension), to obtain a spatial correlation vector with a size of 1*(H*W), and then the spatial correlation vector is restored to a feature matrix with a size of 1*H*W, that is, a fourth spatial domain matrix containing global information is obtained; wherein, the matrix operation of the feature vector is actually to make the spatial domain features interact internally by multiplying each feature value with all other feature values, to a certain extent, to quantify the internal relationship of the spatial domain feature values, and the sum along a certain dimension is to aggregate the information, to summarize the feature internal interaction calculated before into a specific feature value, so that the feature has a global characteristic;

[0072] Step 3.4, weight acquisition: the fourth spatial domain matrix is processed using a Sigmoid activation function to obtain a third attention weight matrix with a size of 1*H*W, the third attention weight matrix is copied to obtain a fourth attention weight matrix with a size of C*H*W;

[0073] Step 3.5, weighted calculation: the fourth attention weight matrix is multiplied with the initial feature matrix element by element to obtain a second feature matrix with a size of C*H*W and spatial correlation attention weighting.

[0074] Step 4, the initial feature matrix is extracted and processed for feature channel correlation to obtain an attention weight in the channel domain, and a third feature matrix is obtained after channel correlation attention weighting. Convolutional neural networks generally extract image information in spatial and channel domains. In general research, channel domain features are considered to be the features of different targets in the image, and the correlation between different channel features can extract target relationship information in the image. Therefore, the specific implementation of the feature channel correlation extraction and processing in this embodiment is as follows:

[0075] Step 4.1, extract channel domain information: two-dimensional average pooling is performed on the spatial domain of the initial feature matrix, and a dimension with a size of 1 is removed, so that the initial feature matrix with a size of C*H*W is converted into a second feature vector with a size of C*1; This can focus the feature processing on the channel domain, excluding the interference and possible additional parameters introduced by the spatial domain information;

[0076] Step 4.2, acquisition and processing of channel correlation information: determine the channel correlation attention width C according to the number of feature channels, and adaptively confirm the parameter k according to the channel width C, use a convolution kernel with a size of k to perform one-dimensional convolution operation on the second feature vector without changing the channel number, to obtain a third feature vector with a size of C*1 and containing channel interaction information; wherein, the one-dimensional convolution operation with a size of k is actually to calculate the interaction between other channels with a range of k around each channel in the feature vector, to extract the channel correlation information in the limited perception domain;

[0077] Step 4.3, weight acquisition: the third feature vector is processed using a Sigmoid activation function to obtain a fifth attention weight matrix with a size of C*1, and the fifth attention weight matrix is copied to obtain a sixth attention weight matrix with a size of C*H*W;

[0078] Step 4.4, weighted calculation: the sixth attention weight matrix is multiplied with the initial feature matrix element by element to obtain a third feature matrix with a size of C*H*W and channel correlation attention weighting.

[0079] In step 4.2, the parameter k is adaptively confirmed according to the channel width C, specifically:

[0080]

[0081] In the formula, |·| odd represents the operation of taking the nearest odd number from the calculation result.

[0082] Step 5, the first feature matrix, the second feature matrix, the third feature matrix are weighted and added based on the hyperparameters to obtain a comprehensive enhanced feature matrix, which can be input into the subsequent convolution layer of the network after the backbone network processing. The weighted addition based on the hyperparameters is specifically:

[0083] X = a X1 + b X2 + (1-a-b) X3

[0084] In the formula, X is the enhanced feature matrix after feature enhancement processing, X1 is the first feature matrix, X2 is the second feature matrix, X3 is the third feature matrix, and a and b are hyperparameters.

[0085] Embodiment 2

[0086] Reference Figure 2 The embodiment discloses an image feature enhancement device based on an attention mechanism, which comprises a feature matrix acquisition unit, a feature enhancement unit, a spatial correlation enhancement unit, a channel correlation enhancement unit, and a feature output unit. Specifically:

[0087] The feature matrix acquisition unit is used to acquire the initial feature matrix to be enhanced of the image; the feature enhancement unit is used to perform feature enhancement processing on the initial feature matrix to obtain a first feature matrix weighted by feature enhancement attention; the spatial correlation enhancement unit is used to extract and process the feature spatial correlation of the initial feature matrix to obtain a second feature matrix weighted by spatial correlation attention; the channel correlation enhancement unit is used to extract and process the feature channel correlation of the initial feature matrix to obtain a third feature matrix weighted by channel correlation attention; and the feature output unit is used to weight the first feature matrix, the second feature matrix, and the third feature matrix to obtain an enhanced feature matrix and output. The working processes of the units are the same as those of Embodiment 1, and thus will not be described herein.

[0088] Embodiment 3

[0089] The embodiment discloses an image processing method, wherein the image processing is one of image classification, target detection, behavior recognition, or instance segmentation, and a lightweight convolutional neural network is used for image processing. Referring to Figure 3 The process of image processing in the embodiment is as follows:

[0090] Step one, input the target image into the lightweight convolutional neural network;

[0091] Step two, based on the network structure of the lightweight convolutional neural network, use a convolutional block with an N*N convolutional kernel to perform convolutional processing on the image feature matrix of the target image;

[0092] Step three, in the repeated convolutional process of the network, the feature enhancement device of embodiment 2 is added in each convolutional block, and then the attention is used to guide the feature construction in the process of extracting the feature matrix in the convolutional layer, that is, for each convolutional block, the image feature matrix output by the convolutional block itself is the input information of the feature matrix acquisition unit in the feature enhancement device in the convolutional block, and the enhanced feature matrix output by the feature output unit in the feature enhancement device in the convolutional block is the input of the next convolutional block, and the same is true for the subsequent convolutional blocks, until the last enhanced feature matrix is output by the feature enhancement device in the last convolutional block;

[0093] Step four, based on the network structure of the lightweight convolutional neural network, the feature matrix output by the last convolutional block and the feature enhancement device is integrated and processed in sequence using 1*1 convolutional operation and pooling operation to improve the semantic expressiveness of the features, and the integration process is as follows: the last enhanced feature matrix obtained in step three is subjected to convolutional operation with a convolutional kernel size of 1*1 to further extract information from the features; and then the feature matrix output by the 1*1 convolution is subjected to two-dimensional pooling, so that the feature matrix with a size of C*H*W is changed into a feature vector with a size of C*1*1;

[0094] Step five, the feature vector is subjected to final processing to calculate and convert it into a format required by a task, and the result of target behavior recognition is output. For example, in the case of target behavior recognition, the specific implementation process of step five is as follows:

[0095] The feature vector obtained in step four is input into a fully connected layer. The input feature vector has a size of C*1, and the output probability distribution vector has a size of C’*1 (the value of C’ is equal to the total number of behavior recognition classes). The item with the maximum value in the output probability distribution vector is taken as the prediction result of the image behavior label, and the target behavior recognition is completed.

[0096] The above only describes the preferred embodiments of the present application, and does not limit the patent scope of the present application, and any equivalent structural transformation made based on the inventive concept of the present application, or direct / indirect application in other related technical fields is included in the patent protection scope of the present application.

Claims

1. An attention mechanism-based feature enhancement method, characterized in that, The method comprises the following steps: Step 1, obtaining an initial feature matrix of an image to be enhanced; Step 2, performing feature enhancement processing on the initial feature matrix to obtain a first feature matrix weighted by attention of feature enhancement, specifically comprising: Step 2.1, extracting spatial domain information: one-dimensional average pooling is performed on the channel domain of the initial feature matrix, so that the initial feature matrix with a size of is converted into a first spatial domain matrix with a size of ; Step 2.2, magnifying the representation difference ratio of the feature: performing an element-wise square operation on the first spatial domain matrix of size to magnify the representation difference of the spatial domain feature, to obtain a second spatial domain matrix; Step 2.3, weight acquisition: the second spatial domain matrix is processed by using a Sigmoid activation function to obtain a first attention weight matrix with a size of The first attention weight matrix is copied to obtain a second attention weight matrix with a size of ​ Step 2.4, Weighted calculation: Element-wise multiply the second attention weight matrix with the initial feature matrix to get a new feature matrix with size and the first feature matrix after feature-enhanced attention weighting; Step 3, extracting and processing spatial correlation of features of the initial feature matrix to obtain a second feature matrix weighted by attention of spatial correlation; Step 4, extracting and processing channel correlation of features of the initial feature matrix to obtain a third feature matrix weighted by attention of channel correlation; Step 5, weighting the first feature matrix, the second feature matrix and the third feature matrix to obtain an enhanced feature matrix.

2. The attention mechanism based feature enhancement method of claim 1, wherein, Step 3 specifically comprises: Step 3.1, extracting spatial domain information: one-dimensional average pooling is performed on the channel domain of the initial feature matrix, so that the initial feature matrix with a size of is converted into a third spatial domain matrix with a size of ; Step 3.2, Matrix Vectorization: For a matrix of size... The first spatial domain matrix is ​​vectorized to make... The third space domain matrix of size is transformed into The first eigenvector of size; Step 3.3, Obtaining and processing the correlation matrix of the feature space: Perform matrix multiplication between the transposed first eigenvector and itself to obtain... The feature space correlation matrix of size is summed along any dimension of the feature space correlation matrix to obtain a matrix of size. The spatial correlation vector, and then the spatial correlation vector is restored to The characteristic matrix of the size is obtained, which is the fourth spatial domain matrix containing global information; Step 3.4, weight acquisition: the fourth spatial domain matrix is processed by using a Sigmoid activation function to obtain a third attention weight matrix with a size of The third attention weight matrix is copied to obtain a fourth attention weight matrix with a size of . Step 3.5, Weighted calculation: Element-wise multiply the fourth attention weight matrix with the initial feature matrix to get a second feature matrix with size and spatial correlation attention weighted.

3. The attention mechanism based feature enhancement method of claim 1, wherein, Step 4 specifically comprises: Step 4.1, extract channel domain information: two-dimensional average pooling is performed on the spatial domain of the initial feature matrix, and a dimension with a size of 1 is removed, so that the initial feature matrix with a size of is converted into a second feature vector with a size of ; Step 4.2, acquisition and processing of channel correlation information: determining channel correlation attention width according to feature channel number C , and determining channel correlation attention width according to channel width C Adaptively confirming parameters k , performing one-dimensional convolution operation on the second feature vector using a convolution kernel with a size of k , to obtain a third feature vector with a size of and containing channel interaction information Step 4.3, weight acquisition: the third feature vector is processed by using a Sigmoid activation function to obtain a fifth attention weight matrix with a size of The fifth attention weight matrix is copied to obtain a sixth attention weight matrix with a size of ​ Step 4.4, Weighted calculation: Element-wise multiply the sixth attention weight matrix with the initial feature matrix to get a third feature matrix of size and channel correlation attention weighted third feature matrix.

4. The attention mechanism based feature enhancement method of claim 3, wherein, In step 4.2, the channel width C Adaptive confirmation of parameters k , in particular: In the formula, represents an operation of taking the nearest odd number from the calculation result.

5. The attention mechanism based feature enhancement method of claim 1, wherein, In Step 5, the enhanced feature matrix is specifically: In the formula, X is a strengthened feature matrix after feature strengthening processing, X1 is a first feature matrix, X2 is a second feature matrix, X3 is a third feature matrix, a , b is a hyperparameter.

6. An attention mechanism-based feature enhancement device, characterized by, The method for image feature enhancement according to any one of claims 1 to 5, wherein the feature enhancement device comprises: a feature matrix acquisition unit configured to acquire an initial feature matrix of an image to be enhanced; a feature enhancement unit configured to perform feature enhancement processing on the initial feature matrix to obtain a first feature matrix weighted by attention of feature enhancement; a spatial correlation enhancement unit configured to extract and process spatial correlation of features of the initial feature matrix to obtain a second feature matrix weighted by attention of spatial correlation; a channel correlation enhancement unit configured to extract and process channel correlation of features of the initial feature matrix to obtain a third feature matrix weighted by attention of channel correlation; a feature output unit configured to weight the first feature matrix, the second feature matrix and the third feature matrix to obtain an enhanced feature matrix and output the enhanced feature matrix.

7. An image processing method, the image processing being image classification or object detection or action recognition or instance segmentation, characterized in that, The method for image processing using a lightweight convolutional neural network, which comprises the following steps: Step 1, inputting a target image into the lightweight convolutional neural network; Step two, based on the network structure of lightweight convolutional neural network, using the convolution block with Convolution kernel to the image feature matrix of the target image Step 3, adding the feature enhancement device according to claim 6 to each convolutional block in the repeated convolution process of the network, so as to guide feature construction by attention in the process of extracting the feature matrix in the convolutional layer; Step four, based on the network structure of lightweight convolutional neural network, the last convolutional block and the feature matrix output by the feature enhancement device are sequentially processed using Convolution operation and pooling operation are used to improve the semantic expression of features; Step 5, performing final processing on the integrated feature matrix to make it to the format required by the task for calculation and conversion, and outputting the result of image processing.

Images