Object detection method based on attention mechanism
By constructing a convolutional neural network of feature extraction, attention mechanism, feature enhancement and recognition positioning module, the problems of poor object detection effect and slow detection speed in complex backgrounds are solved, and real-time accurate object detection in intelligent driving systems are achieved.
Patent Information
- Application Number
- CN202210339874.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-01
- Publication Date
- 2024-12-17
- Estimated Expiration
- 2042-04-01
AI Technical Summary
The existing object detection methods have poor detection effects in complex contexts, complex networks and unfast speeds, making them difficult to apply in real-time scenarios.
A convolutional neural network consisting of feature extraction module, attention mechanism module, feature enhancement module and identification and positioning module is built. Through the attention mechanism, the feature extraction and detection capabilities are enhanced, and the network structure is simplified to improve detection speed.
In a complex context, precise identification of target detection is achieved, detection accuracy is improved and real-time detection capabilities are achieved, and it is suitable for intelligent driving systems.
Smart Images
Figure CN114882205B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of image processing technology, relates to image detection, and in particular to a target detection method based on an attention mechanism. Background Art
[0002] A large number of existing target detection methods have achieved very high scores in public datasets, but there are still many challenges in target detection in real-world tasks, such as poor detection of targets in complex backgrounds. For example, on actual roads, real-time detection of pedestrians and vehicles by intelligent driving systems will help drivers drive more safely. However, the background of real-world situations is very complex. For example, bad weather such as rainy and snowy days will affect the detection of pedestrians and vehicles, and the detection of pedestrians and vehicles is also poor in the evening and at night. The detection of pedestrians and vehicles in different scenarios such as highways or urban roads will also be affected.
[0003] Patent CN 112163580 A discloses a small target detection algorithm based on the attention mechanism. The algorithm adds an attention mechanism during the detection process, so that the attention mechanism is applied to the feature maps of different channels, enhancing the channels that are beneficial to recognition and suppressing the channels that are useless for recognition. The method first inputs the image into the RPN network to extract the pre-selected area; inputs the acquired area into the FPN to extract multi-scale features; inputs the multi-scale features into the attention mechanism module; in the attention mechanism module, the weights of the feature channels are calculated through two fully connected layers and a Sigmoid function; the obtained feature map is input into the ROI Align layer; classification and regression are performed according to the obtained feature map to obtain the detection result. Although this method can improve the accuracy of target detection, the method still has the following shortcomings: the performance of the feature extraction network is general, so that the effect of adding the attention mechanism is not obvious, resulting in poor target detection effect under actual complex backgrounds.
[0004] Han Hu et al. published a method for image object detection based on attention mechanism in their paper Relation Networks for Object Detection (The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018). This method introduces an object relationship module to characterize the relationship between objects, and uses such attention to improve the effect of object detection. In the detection process, this method can optimize the detection effect by utilizing the relationship between objects in the image, or the image content. This relationship includes both relative position relationship and image feature relationship. That is, first input the image into the main feature extraction network; obtain ROI and features (RPN network is included in it); perform border regression and object classification based on ROI, add attention mechanism after the fully connected layer of this network; perform non-maximum processing and remove duplicate frames. Although this network method can improve the detection accuracy of the target, the method still has the disadvantage that the large number of parameters leads to slow detection speed, so it cannot be used in fast detection scenarios. Summary of the invention
[0005] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a target detection method based on an attention mechanism to solve the problems of complex network, slow speed and poor target detection effect in complex backgrounds in existing target detection methods, and to detect targets in complex backgrounds in real time.
[0006] In order to achieve the above object, the technical solution adopted by the present invention is:
[0007] Object detection methods based on attention mechanisms include:
[0008] Step 1: Build a convolutional neural network
[0009] The convolutional neural network is composed of a feature extraction module, an attention mechanism module, a feature enhancement module, and a recognition and positioning module connected in sequence; the feature extraction module is used to extract target features, the attention mechanism module is used to change the feature weights of different channels, the feature enhancement module is used to enhance the extracted target features, and the recognition and positioning module is used to identify and locate the enhanced target features;
[0010] Step 2: Generate target training set
[0011] The target training set consists of a plurality of images to be detected containing targets, and the targets in each image to be detected are marked, and the marking is to mark the coordinates of each vertex of each circumscribed rectangular box where the target is located and the category it represents;
[0012] Step 3: training the convolutional neural network
[0013] Step 4: Detect the target to be detected
[0014] The image containing the target to be detected is input into the trained convolutional neural network, and the detection results of the category of the target to be detected in the image and the position of each circumscribed rectangular box where the target is located are output.
[0015] Compared with the prior art, the present invention has the following beneficial effects:
[0016] First, since the present invention constructs a target detection network composed of a feature extraction module, an attention mechanism module, a feature enhancement module, and an identification and positioning module, the attention mechanism module is mainly aimed at solving scenes where the background is complex and the detection network has difficulty distinguishing between the detection target and the background. By adding the attention mechanism, the weight of the detection target of the feature extraction module output result is increased, and the weight of the background of the feature extraction module output result is reduced. Ultimately, the problem that it is difficult to effectively extract the features of the object to be detected in a complex background in the existing target detection method, resulting in low final detection accuracy, is overcome, so that the present invention can accurately identify the target to be detected in a complex background.
[0017] Second, since the present invention constructs an end-to-end target detection network, the overall network structure is very simple, but the feature extraction capability is strong and the detection speed is fast due to the small amount of calculation, which overcomes the problem of slow detection speed of existing target detection methods, so that the present invention can be applied to intelligent driving systems to accurately identify vehicles and pedestrians in real time and realize the function of real-time target detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 It is a flow chart of the present invention.
[0019] Figure 2 Schematic diagram of the convolutional neural network structure of the present invention
[0020] Figure 3 This is a structural diagram of the attention mechanism module in the present invention.
[0021] Figure 4 This is the detection result without adding the attention module.
[0022] Figure 5 Detection results after adding the attention module. DETAILED DESCRIPTION
[0023] The embodiments of the present invention are described in detail below with reference to the accompanying drawings and examples.
[0024] The present invention is a target detection method based on attention mechanism. It extracts target features by constructing a convolutional neural network, and uses the attention mechanism to change the feature weights of different channels. Then, the extracted target features are enhanced. Finally, the output vector is matched with the category and position, and the target is identified and located, so that each target in the image can be accurately detected.
[0025] refer to Figure 1 , the present invention specifically comprises the following steps:
[0026] Step 1: Build a convolutional neural network:
[0027] refer to Figure 2 The convolutional neural network of the present invention is composed of a feature extraction module, an attention mechanism module, a feature enhancement module, and a recognition and positioning module connected in sequence. Among them:
[0028] The feature extraction module is used to extract target features. The module has 9 layers, and its structure is as follows: Focus layer → first convolution layer → first CSP layer → second convolution layer → second CSP layer → third convolution layer → third CSP layer → fourth convolution layer → pooling layer. The parameters of each layer are set as follows:
[0029] The number of convolution kernels in the first to fourth convolution layers is set to 64, 128, 256, and 512 respectively, the size of the convolution kernel is set to 3×3, the step size is set to 2, and each convolution layer needs to perform convolution operation, batch normalization processing, and ReLU activation processing.
[0030] The Focus layer needs to perform a downsampling and convolution operation. During downsampling, the width and height will be halved, the number of channels will be increased to four times the original, and the convolution kernel size of the convolution operation is 1×1, with a step size of 1.
[0031] The CSP layer has two operations. The upper path is a convolution operation and then a residual structure. The lower path is a convolution operation. The two paths are concatenated and then convolved. The size of each convolution kernel is set to 3×3 and the step size is set to 2.
[0032] The pooling layer has four channels, one of which is the input channel, and the sizes of the pooling area kernels of the other three channels are 5×5, 9×9, and 13×13, respectively. The step sizes are all set to 2, and the maximum pooling method is adopted. The four-channel structure is spliced and then convolution operation is performed. The convolution kernel size is 3×3 and the step size is 2.
[0033] The function of the input volume of each convolutional layer is as follows:
[0034] V in=H 1 *W 1 *D 1
[0035] Among them, H 1 , W 1 and D 1 is the input height, width and number of output channels.
[0036] The function of the output volume of each convolutional layer is:
[0037] V out =H 2 *W 2 *D 2
[0038] Among them, H 2 , W 2 and D 2 is the output height, width and number of output channels.
[0039] H 2 , W 2 and D 2 The formula is as follows:
[0040] H 2 =(H 1 -F+2*P) / S+1
[0041] W 2 =(W 1 -F+2*P) / S+1
[0042] D 2 =K
[0043] Among them, F is the size of the convolution kernel, S is the step size, P is the size of zero padding, and K is the number of convolution kernels.
[0044] The function of batch normalization processing of the convolutional layer is as follows:
[0045]
[0046] Among them, r i is the i-th feature weight before updating, s i represents the updated i-th feature weight, and C is the number of channels.
[0047] This method can achieve the downsampling effect well by using the Focus layer, and can reduce the number of parameters while widening the number of channels. It can extract the features of the detection target in the image well by using multiple convolutional layers and CSP layers. Using the pooling layer instead of the fully connected layer can greatly reduce the amount of data and network parameters, thereby improving the detection speed.
[0048] The attention mechanism module is used to change the feature weights of different channels. There are 3 layers in total. Figure 2 and Figure 3 , its structure is: global average pooling layer → weight-sharing one-dimensional convolution layer → Sigmoid activation layer; and the size of the pooling area kernel is set to 2×2, the step size is set to 2, the convolution kernel size of the one-dimensional convolution layer is k, and the padding is k / 2.
[0049] The function of the global average pooling layer is:
[0050]
[0051] Among them, z i represents the i-th feature weight of the global average pooling layer output, i = 1, 2, ..., C, C represents the total dimension of X, that is, the number of channels, C = 256, F global represents the global average pooling operation, X represents a set of feature maps extracted from the feature extraction module and input to the global average pooling layer, H and W represent the height and width of X respectively, m and n represent the horizontal axis coordinate and the vertical axis coordinate respectively. The coordinate system takes the upper left corner of the feature map as the origin, the horizontal right direction is the positive direction of the horizontal axis, and the vertical downward direction is the positive direction of the vertical axis. i (m,n) represents the value of the point with coordinates (m,n) in the i-th feature map in X.
[0052] The weight-sharing one-dimensional convolutional layer is different from the convolutional layer in the feature extraction module. Its convolution kernel size k represents the coverage of local cross-channel interactions. The relationship between k and C is:
[0053]
[0054] Among them, ||odd means that k can only be an odd number, γ and b are two parameters used to control the number of channels C and the size of the convolution kernel k. This method sets γ and b to 2 and 1 respectively to change the ratio between the number of channels C and the size of the convolution kernel k.
[0055] The function of the Sigmoid activation layer is:
[0056]
[0057] Among them, x is the input and S(x) is the output. By using this function, the nonlinearity of the network can be well increased, and the output range can be controlled between (0,1), making the optimization stable. In addition, this function is a continuous function and is easy to calculate.
[0058] In actual scenarios, different backgrounds will have a certain impact on target detection. For example, the accuracy of target detection in urban streets with many pedestrians and vehicles will be reduced. Target detection on rainy or snowy days will be affected by the weather. Detection in the evening or at night will be more difficult than target detection during the day. In addition, the real scene is very complex. These influences lead to poor performance of existing target detection methods. However, this method can effectively change the weight ratio of each channel in the feature map output by the feature extraction network by using the attention mechanism, enhance the feature weight of the target to be detected, reduce the feature weight of other backgrounds, make the extracted features more effective, and greatly improve the detection ability of the overall network. Therefore, this method is particularly suitable for target detection in complex backgrounds.
[0059] The feature enhancement module is used to enhance the extracted target features. It has a total of 14 layers, and the structure is as follows: the fifth convolution layer → the first upsampling layer → the first channel splicing layer → the fourth CSP layer → the sixth convolution layer → the second upsampling layer → the second channel splicing layer → the fifth CSP layer → the seventh convolution layer → the third channel splicing layer → the sixth CSP layer → the eighth convolution layer → the fourth channel splicing layer → the seventh CSP layer.
[0060] Among them, the convolution kernel size of the convolution layer is 3×3, and the step size is 2. The structure of the CSP layer is the same as the setting of the first step feature extraction module. The inputs of the first to fourth channel splicing layers are the first upsampling layer and the third CSP layer, the second upsampling layer and the second CSP layer, the sixth convolution layer and the seventh convolution layer, and the fifth convolution layer and the eighth convolution layer respectively; and the convolution kernel size of the convolution layer is set to 2×2, and the step size is set to 2.
[0061] This method uses a feature enhancement module with a relatively simple structure and less computation. However, the module uses convolutional layers, channel splicing layers, and CSP layers multiple times to fuse multi-scale feature information, thereby better enhancing the feature map output by the attention mechanism. It is simpler than the current target detection network but has a significant feature enhancement effect.
[0062] The recognition and positioning module is used to recognize and locate the enhanced target features. The module has a total of 3 layers, and its structure consists of three convolutional layers, namely: the ninth convolutional layer, the tenth convolutional layer, and the eleventh convolutional layer; the inputs of the ninth convolutional layer, the tenth convolutional layer, and the eleventh convolutional layer are connected to the outputs of the fifth CSP layer, the sixth CSP layer, and the seventh CSP layer respectively; and the number of convolution kernels in the ninth, tenth, and eleventh convolutional layers are set to 512, 768, and 1024 respectively, the size of the convolution kernel is set to 1×1, and the step size is set to 1.
[0063] The feature extraction module, attention mechanism module, feature enhancement module, recognition and positioning module are connected in sequence to form a convolutional neural network.
[0064] Step 2: Generate target training set:
[0065] The target training set of the present invention is composed of a plurality of images to be detected containing targets, and the targets in each image to be detected are annotated, that is, the coordinates of each vertex of each circumscribed rectangular box where the target is located and the category it represents are annotated. For example, a picture can be intercepted every 10 frames from a continuously shot video containing the target to be detected as a picture to be detected, and a total of 10,000 pictures can be intercepted, and the acquired pictures can be annotated. For example, the target category to be detected in the picture can be annotated with labelImg software, and the coordinates and category of the annotation box can be determined, that is, the target training set is formed.
[0066] In one embodiment of the present invention, the target training set is generated by the following method:
[0067] The first step is to form a picture data set with at least 10,000 pictures of vehicles or pedestrians to be detected. Each picture to be detected is captured every 10 frames from the continuously shot videos containing vehicles or pedestrians, and a total of 10,000 pictures are captured;
[0068] The second step is to label the targets in each image in the image dataset. Use labelImg software to label the bus, truck, car, and person in the image, determine the coordinates and categories of the label boxes, and form a target training set.
[0069] Step 3: Train the convolutional neural network:
[0070] The training set is input into the convolutional neural network, and the weights of the convolutional neural network are updated using the gradient descent method until the Loss value drops below 3.0 to obtain a trained convolutional neural network.
[0071] The Loss function of the present invention is defined as:
[0072] Loss=Loss box +Loss obj +Loss cls
[0073] Among them, Loss box To evaluate the position loss of the target box and the predicted box, Loss obj To evaluate the confidence loss of the target box and the predicted box, Loss cls To evaluate the category loss of the target box and the predicted box.
[0074] Loss box The formula is as follows:
[0075]
[0076] Among them, IOU is the intersection-union ratio of the predicted box and the real box, Distance_2 is the Euclidean distance between the two center points of the predicted box and the real box, Distance_C is the diagonal distance between the predicted box and the real box, and v is a parameter to measure the consistency of the aspect ratio, which is defined as:
[0077]
[0078]
[0079] Among them, A and B are the areas of the predicted box and the real box respectively, and w gt and h gt Indicates the width and height of the real box, w p and w p Represents the width and height of the prediction box.
[0080] Loss obj The formula is:
[0081]
[0082] in, Indicates whether it is a positive sample, with a value of 0 or 1. Indicates traversing all prediction boxes, S*S=20*20+40*40+80*80, B=4, Indicates whether it is a negative sample, with a value of 0 or 1. represents the sample value, C i Represents the predicted value.
[0083] Loss cls The formula is:
[0084]
[0085] Where i and j are the indexes of anchors in a mini-batch, i increases from 0 to S*S, j increases from 0 to the value of B, and p i is the probability that the anchor is predicted as the target. When the anchor is the target, is 1, otherwise it is 0. Indicates traversing all prediction boxes, S*S=20*20+40*40+80*80, B=4, Indicates whether it is a positive sample, the value is 0 or 1, Represents the true probability value of the target, p i (c) represents the target prediction probability value.
[0086] Step 4: Detect the target to be detected:
[0087] Each picture containing a vehicle or pedestrian is input into the trained convolutional neural network in turn, and the detection results of the category of the vehicle or pedestrian in the picture and the position of each circumscribed rectangular box where the target is located are output. Figure 4 and Figure 5 It is the specific vehicle or pedestrian detection result, where car represents the detected car, truck represents the detected truck, and the number represents the possibility probability. Figure 4 and Figure 5 The following are the actual detection effects of the target detection method without and with attention mechanism. It can be seen that this method can detect target vehicles or pedestrians well in complex backgrounds, and the detection accuracy is much higher than that of the network without attention mechanism, which highly proves that this network has high target detection performance in complex backgrounds.
Claims
1. Object detection method based on attention mechanism, It is characterized in that The steps include: Step 1: Build a convolutional neural network The convolutional neural network is composed of a feature extraction module, an attention mechanism module, a feature enhancement module, and an identification and positioning module connected in sequence; the feature extraction module is used to extract the features of the target to be detected, the attention mechanism module is used to change the feature weights of different channels, the feature enhancement module is used to enhance the extracted target features, and the identification and positioning module is used to identify and locate the enhanced target features; The feature extraction module has 9 layers in total, and its structure is as follows: Focus layer → first convolution layer → first CSP layer → second convolution layer → second CSP layer → third convolution layer → third CSP layer → fourth convolution layer → pooling layer; Set the parameters of each layer to: The number of convolution kernels in the first to fourth convolution layers is set to 64, 128, 256, and 512 respectively, the size of the convolution kernel is set to 3×3, the step size is set to 2, and each convolution layer needs to perform convolution operation, batch normalization processing, and ReLU activation function processing; The Focus layer performs a downsampling and convolution operation. When downsampling, the width and height are reduced to half of the original, and the number of channels is increased to four times the original. The convolution kernel size of the convolution operation is 1×1, and the stride is 1; The CSP layer has two operations. The upper path is a convolution operation and then a residual structure. The lower path is a convolution operation. The two paths are concatenated and then convolved. The size of each convolution kernel is set to 3×3 and the step size is set to 2. The pooling layer has four channels, one of which is the input channel, and the sizes of the pooling area kernels of the other three channels are 5×5, 9×9, and 13×13 respectively. The step sizes are all set to 2, and the maximum pooling method is used. The four-channel structure is spliced and then convolution operation is performed. The convolution kernel size is 3×3 and the step size is 2; The attention mechanism module has 3 layers, and its structure is as follows: global average pooling layer → weight-sharing one-dimensional convolution layer → sigmoid activation layer; the size of the pooling area kernel is set to 2×2, and the step size is set to 2; different from the convolution layer of the feature extraction module, the convolution kernel size of the weight-sharing one-dimensional convolution layer is k, and the padding is k / 2; The function of the global average pooling layer is: Among them, z i represents the i-th feature weight of the global average pooling layer output, i = 1, 2, ..., C, C represents the total dimension of X, that is, the number of channels, C = 256, F global represents the global average pooling operation, X represents a set of feature maps extracted from the feature extraction module and input to the global average pooling layer, H and W represent the height and width of X respectively, and a coordinate system is established with the upper left corner of the feature map as the origin, horizontal to the right as the positive direction of the horizontal axis, and vertically downward as the positive direction of the vertical axis. m and n represent the horizontal and vertical coordinates respectively, and x i (m,n) represents the value of the point with coordinates (m,n) in the i-th feature map in X; The weight-sharing one-dimensional convolutional layer is different from the convolutional layer in the feature extraction module. Its convolution kernel size k represents the coverage of local cross-channel interactions. The relationship between k and C is: Among them, |||odd means that k can only be an odd number, γ and b are two parameters used to control C and k, which are used to change the ratio between C and k; The function of the Sigmoid activation layer is: Among them, x is the input and S(x) is the output; The feature enhancement module has a total of 14 layers, and its structure is as follows: the fifth convolution layer→the first upsampling layer→the first channel splicing layer→the fourth CSP layer→the sixth convolution layer→the second upsampling layer→the second channel splicing layer→the fifth CSP layer→the seventh convolution layer→the third channel splicing layer→the sixth CSP layer→the eighth convolution layer→the fourth channel splicing layer→the seventh CSP layer; wherein the convolution kernel size of the convolution layer is 3×3, and the step size is 2, the structure of the CSP layer is the same as the setting of the first step feature extraction module, and the inputs of the first to fourth channel splicing layers are the first upsampling layer and the third CSP layer, the second upsampling layer and the second CSP layer, the sixth convolution layer and the seventh convolution layer, the fifth convolution layer and the eighth convolution layer; and the convolution kernel size of the convolution layer is set to 2×2, and the step size is set to 2; The identification and positioning module has 3 layers in total, and its structure is three convolutional layers, namely: the ninth convolutional layer, the tenth convolutional layer, and the eleventh convolutional layer; the inputs of the ninth convolutional layer, the tenth convolutional layer, and the eleventh convolutional layer are connected to the outputs of the fifth CSP layer, the sixth CSP layer, and the seventh CSP layer respectively; and the number of convolution kernels in the ninth, tenth, and eleventh convolutional layers is set to 512, 768, and 1024 respectively, the size of the convolution kernel is set to 1×1, and the step size is set to 1; Step 2: Generate target training set The target training set consists of a plurality of images to be detected containing targets, and the targets in each image to be detected are marked, and the marking is to mark the coordinates of each vertex of each circumscribed rectangular box where the target is located and the category it represents; Step 3, training the convolutional neural network; Step 4: Use the trained convolutional neural network to detect the target to be detected; The image containing the target to be detected is input into the trained convolutional neural network, and the detection results of the category of the target to be detected in the image and the position of each circumscribed rectangular box where the target is located are output.
2. According to the object detection method based on the attention mechanism of claim 1, It is characterized in that The batch normalization process is to normalize each batch of data, and the processing method used is min-batch SGD; the ReLU activation function is used to increase the nonlinearity of the convolutional neural network and overcome the problem of gradient disappearance. The function formula is: f(x)=max(0,x) In the formula, x is the input feature information, and f(x) is the output feature information.
3. According to the object detection method based on the attention mechanism of claim 1, It is characterized in that The function of the input volume of each convolutional layer is as follows: V in =H 1 *W 1 *D 1 Among them, H 1 , W 1 and D 1 is the input height, width and number of output channels; The function of the output volume of each convolutional layer is: V out =H 2 *W 2 *D 2 Among them, H 2 , W 2 and D 2 is the output height, width and number of output channels; H 2 , W 2 and D 2 The formula is as follows: H 2 =(H 1 -F+2*P) / S+1 W 2 =(W 1 -F+2*P) / S+1 D 2 =K Among them, F is the size of the convolution kernel, S is the step size, P is the size of zero padding, and K is the number of convolution kernels; The function of the batch normalization process is as follows: Among them, r i is the i-th feature weight before updating, S i Represents the updated i-th feature weight.