Image detection method and device based on Faster-RCNN
By integrating a Transformer module with CNN in Faster-RCNN, the model effectively captures global information and long-distance dependencies, improving object detection accuracy and recognition rates.
Patent Information
- Application Number
- CN202411101415.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2024-12-03
AI Technical Summary
Current deep learning models, such as Faster-RCNN, struggle with capturing global information features and long-distance dependencies in image recognition due to the limitations of CNNs, which primarily retain local features and fail to effectively utilize long-range connections.
Integrate a Transformer module with a CNN network to enhance feature extraction, incorporating a region candidate network and area of interest pooling layer to improve object localization and recognition accuracy.
The integrated network structure captures both local and global features, enhancing recognition rates and accurately locating objects without increasing the size of CNN convolution kernels.
Smart Images

Figure CN119068174A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of neural network image detection, and more specifically, to an image detection method and device based on Faster-RCNN. Background Art
[0002] With the development of deep learning technology, models such as VGG16 and Alexnet have been widely used in the field of computer vision. The current deep learning technology has achieved a very high accuracy in image recognition, but it is still insufficient in the feature preprocessing of global information features of feature maps. For example, in the feature map extraction of faster-RCNN, CNN with dilated convolution is generally used to extract feature maps, but CNN can generally only retain local features and it is difficult to capture global information. At the same time, two adjacent pixels of an image sometimes have a certain connection, and the current network structure is difficult to capture long-distance dependencies. Summary of the invention
[0003] The purpose of the present invention is to propose an image detection method and device based on Faster-RCNN, so as to achieve better extraction of global information features, improve recognition rate and accurately locate objects.
[0004] To achieve the above objectives, in a first aspect, the present invention proposes an image detection method based on Faster-RCNN, comprising:
[0005] Build an image detection model based on Faster-RCNN, including:
[0006] Construct a feature extraction network that integrates the Transformer module and the CNN network, wherein the feature extraction network is used to extract features from the input image and generate a feature map;
[0007] Constructing a region candidate network, wherein the region candidate network is used to generate a candidate box for locating an object;
[0008] Constructing an area of interest pooling layer and a fully connected layer, wherein the area of interest pooling layer is used to collect candidate boxes generated by the area candidate network, and extract feature maps from the feature extraction network to generate candidate box feature maps, and the fully connected layer is used to calculate specific categories using the candidate box feature maps, and obtain the final precise position of the detection box through bounding box regression;
[0009] Using a pre-constructed training data set to train the constructed image detection model;
[0010] The trained image detection model is used to perform target detection on the image to be detected.
[0011] Optionally, the feature extraction network includes: a 3*3 one-dimensional convolution layer, a 64*64 one-dimensional convolution layer, a 2*2 maximum pooling layer, a fully connected layer, a first normalization layer, a multi-head attention mechanism layer, a second normalization layer and a multi-layer perceptron layer connected in sequence;
[0012] Among them, a 3*3 one-dimensional convolution layer, a 64*64 one-dimensional convolution layer, a 2*2 maximum pooling layer and a fully connected layer constitute the CNN network; the first normalization layer, the multi-head attention mechanism layer, the second normalization layer and the multi-layer perceptron layer constitute the Transformer module.
[0013] Optionally, both the 3*3 one-dimensional convolution and the 64*64 one-dimensional convolution use the ReLU activation function.
[0014] Optionally, the region candidate network is used to generate a candidate box for accurately locating an object using a kmeans algorithm.
[0015] Optionally, the regional candidate network includes:
[0016] An anchor box generation module, used to generate an anchor box;
[0017] The anchor box extraction module is used to classify the generated anchor boxes using a softmax classifier and extract the anchor boxes with targets;
[0018] The bounding box regression module regresses the offset between the anchor box with the target and the annotated position of the real target to correct the anchor box, improve the accuracy of locating the object and output the candidate box.
[0019] In a second aspect, the present invention proposes an image detection device based on Faster-RCNN, comprising:
[0020] A feature extraction network, wherein the feature extraction network uses a network structure that integrates a Transformer module and a CNN network to extract features from an input image and generate a feature map;
[0021] Region proposal network, used to generate candidate boxes for locating objects;
[0022] The region of interest pooling layer is used to collect the candidate boxes generated by the region candidate module and extract the feature map from the feature extraction module to generate a candidate box feature map;
[0023] The fully connected layer is used to calculate the specific category using the candidate box feature map and obtain the final precise position of the detection box through bounding box regression.
[0024] Optionally, the feature extraction network includes: a 3*3 one-dimensional convolution layer, a 64*64 one-dimensional convolution layer, a 2*2 maximum pooling layer, a fully connected layer, a first normalization layer, a multi-head attention mechanism layer, a second normalization layer and a multi-layer perceptron layer connected in sequence;
[0025] Among them, a 3*3 one-dimensional convolution layer, a 64*64 one-dimensional convolution layer, a 2*2 maximum pooling layer and a fully connected layer constitute the CNN network; the first normalization layer, the multi-head attention mechanism layer, the second normalization layer and the multi-layer perceptron layer constitute the Transformer module.
[0026] Optionally, both the 3*3 one-dimensional convolution and the 64*64 one-dimensional convolution use the ReLU activation function.
[0027] Optionally, the region candidate network is used to generate a candidate box for accurately locating an object using a kmeans algorithm.
[0028] Optionally, the regional candidate network includes:
[0029] An anchor box generation module, used to generate an anchor box;
[0030] The anchor box extraction module is used to classify the generated anchor boxes using a softmax classifier and extract the anchor boxes with targets;
[0031] The bounding box regression module regresses the offset between the anchor box with the target and the annotated position of the real target to correct the anchor box, improve the accuracy of locating the object and output the candidate box.
[0032] The beneficial effects of the present invention are:
[0033] The present invention uses the parallel model Transformer to fuse the CNN convolutional neural network, and uses this network as a feature map preprocessing network to better perform feature extraction. In the feature map extraction of Faster-RCNN, CNN with dilated convolution is generally used to extract feature maps, but CNN can generally only retain local features and it is difficult to capture global information, while Transformer can capture long-distance dependencies. Therefore, the present invention fuses the two network structures of Transformer and CNN in the feature extraction network, so that this network structure can have the advantages of CNN and Transformer, so as to better perform feature extraction and improve the recognition rate. At the same time, in the RPN network structure, the kmeans algorithm is integrated so that the anchor frame can accurately locate the object.
[0034] The system of the present invention has other characteristics and advantages, which will be apparent from the drawings incorporated herein and the following detailed description, or will be described in detail in the drawings incorporated herein and the following detailed description, which together serve to explain the specific principles of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] The above and other objects, features and advantages of the present invention will become more apparent through a more detailed description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, in which like reference numerals generally represent like components.
[0036] Figure 1 A structural diagram of the existing Faster-RCNN network model is shown.
[0037] Figure 2 A schematic diagram of an image detection model based on Faster-RCNN in one embodiment of the present invention is shown.
[0038] Figure 3 A schematic diagram of the structure of a feature extraction network that integrates a Transformer module and a CNN network in one embodiment of the present invention is shown. DETAILED DESCRIPTION
[0039] The network structure of Faster-RCNN is as follows Figure 1 As shown in the figure, the process of the Faster-RCNN algorithm is input image → generate candidate regions → feature extraction → classification.
[0040] Among them, RCNN_base generally adopts the VGG structure, which consists of a CNN with dilated convolution and a pooling layer. CNN can extract local features of image feature maps well, but it is difficult to capture global features. Two adjacent pixels in an image sometimes have a certain connection. The current network structure is difficult to capture long-distance dependencies, so the Transformer network is integrated into the network structure here to better extract global information features. The VGG network parameters are shown below.
[0041] VGG(
[0042] (features):Sequential(
[0043] (0):Conv2d(3,64,kernel_size=(3,3),stride=(1,1),padding=(1,1))
[0044] (1): ReLU (inplace)
[0045] (2):Conv2d(64,64,kernel_size=(3,3), stride=(1,1),padding=(1,1))
[0046] (3): ReLU (inplace)
[0047] (4):MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=Fal se)
[0048] (5):Conv2d(64,128,kernel_size=(3,3), stride=(1,1), padding=(1,1))
[0049] … ) )
[0052] VGG network structure:
[0053] The first layer input is a 3-channel 2D convolution with a kernel size of 3*3, a stride of 1, and a padding of 1.
[0054] The RELU activation function is a nonlinear activation function that allows values greater than 0 to pass through and closes values less than or equal to 0 to simplify calculations;
[0055] The second layer of the network is a 64-channel 2D convolution with a kernel size of 3*3, a stride of 1, and a padding of 1.
[0056] The third layer of the network is the pooling layer, with a convolution kernel size of 2, a stride of 2, an extended convolution size of 1, and zero padding;
[0057] The fourth layer of the network is a 64-channel 2D convolution with a convolution kernel size of 3*3, a stride of 1, and a padding of 1.
[0058] The limited receptive field of CNN in the existing Faster-RCNN algorithm makes it difficult to capture global information. Some current methods use CNN to increase the receptive field to extract more image information, which will increase the size of the convolution kernel and is not easy to port to low-computing platforms.
[0059] The present invention proposes an image detection method based on Faster-RCNN, which can better extract the global features of the image without increasing the size of the CNN convolution kernel, thereby improving the recognition rate.
[0060] The present invention will be described in more detail below with reference to the accompanying drawings. Although preferred embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.
[0061] Example 1
[0062] This embodiment provides an image detection method based on Faster-RCNN, including:
[0063] Build an image detection model based on Faster-RCNN, including:
[0064] Construct a feature extraction network that integrates the Transformer module and the CNN network, wherein the feature extraction network is used to extract features from the input image and generate a feature map;
[0065] Constructing a region candidate network, wherein the region candidate network is used to generate a candidate box for locating an object;
[0066] Constructing an area of interest pooling layer and a fully connected layer, wherein the area of interest pooling layer is used to collect candidate boxes generated by the area candidate network, and extract feature maps from the feature extraction network to generate candidate box feature maps, and the fully connected layer is used to calculate specific categories using the candidate box feature maps, and obtain the final precise position of the detection box through bounding box regression;
[0067] The image detection model constructed based on Faster-RCNN in this embodiment is as follows Figure 2 shown.
[0068] The structure of the feature extraction network that integrates the Transformer module and the CNN network is as follows Figure 3 As shown, it includes: a 3*3 one-dimensional convolutional layer, a 64*64 one-dimensional convolutional layer, a 2*2 maximum pooling layer, a fully connected layer, a first normalization layer, a multi-head attention mechanism layer, a second normalization layer and a multi-layer perceptron layer connected in sequence;
[0069] Among them, the CNN network (convolutional neural network) is composed of 3*3 one-dimensional convolution (Conv layer), 64*64 one-dimensional convolution, 2*2 maximum pooling layer (Maxpool) and fully connected layer (fc); 3*3 one-dimensional convolution and 64*64 one-dimensional convolution both use ReLU activation function, allowing values greater than 0 to pass through, and closing values less than or equal to 0 to simplify calculation. In addition, 3*3 one-dimensional convolution and 64*64 one-dimensional convolution also use BN (Batch Normalization) algorithm to standardize the input of ReLU activation function before using ReLU activation function, so as to speed up network training and convergence, control gradient explosion and gradient disappearance, and prevent overfitting.
[0070] The first normalization layer (LayerNorm), the multi-head attention mechanism layer (MHSA), the second normalization layer (LayerNorm) and the multi-layer perceptron layer (MLP) constitute the Transformer module ( Figure 2 dotted box part).
[0071] Specifically, Figure 3 As shown in the figure, in the structure of the RCNN_base network (i.e., feature extraction network) after the fusion of Transformer and CNN, the first layer of input is a 3*3 one-dimensional convolution, the second layer is a 64*64 one-dimensional convolution; the third layer is a 2*2 pooling layer with a step size of 2; the input passes through a three-layer convolutional neural network, and then the full connection is used for dimensional transformation. The LayerNorm of the Transformer module (Transformer Block) consists of two parts, Add&Norm. The Add&Norm layer consists of two main bodies, Add and Norm, where Add refers to the residual connection part and Norm refers to the normalized part; MHSA is a multi-head attention mechanism, and MLP (Multilayer Perceptron) is a multi-layer perceptron, which is a feedforward neural network consisting of an input layer, a hidden layer, and an output layer. Each neuron receives the output of the neurons in the previous layer, and processes it through a nonlinear activation function after weighted summation.
[0072] In this embodiment, the region candidate network is used to generate a candidate box for accurately locating an object using a kmeans algorithm.
[0073] The regional candidate networks include:
[0074] An anchor box generation module, used to generate an anchor box;
[0075] The anchor box extraction module is used to classify the generated anchor boxes using a softmax classifier and extract the anchor boxes with targets;
[0076] The bounding box regression module regresses the offset between the anchor box with the target and the annotated position of the real target to correct the anchor box, improve the accuracy of locating the object and output the candidate box.
[0077] Specifically, the RCNN_rpn recommendation network (i.e., region candidate network) generates an anchor box and uses the machine learning kmeans algorithm for regression to accurately locate the object.
[0078] The algorithm of the RCNN_rpn recommendation network mainly includes the following steps:
[0079] 1) Generate Anchor boxes;
[0080] 2) Determine whether the anchor boxes contain foreground or background;
[0081] 3) Regression learning of the position difference between the anchor boxes and the ground truth (true value) to accurately locate the object.
[0082] The RCNN_rpn recommendation network can be summarized into four steps:
[0083] Generate anchors → softmax classifier extracts positive anchors (positive anchor boxes with targets) → bbox regression (bounding box regression) positive anchors generate offsets → generate final proposals (candidate boxes).
[0084] Based on the above, the detection part of the image detection model based on Faster-RCNN in this embodiment mainly includes the following four parts:
[0085] 1) CNN network (i.e. feature extraction network) is used for feature extraction. A set of Conv+relu+pooling layers are used to extract feature maps of the image for subsequent RPN layers and proposal extraction.
[0086] 2) RPN (Region Proposal Network) This network replaces the SelectiveSearch of the previous RCNN version and is used to generate candidate boxes. The task here has two parts: one is classification: judging whether all preset anchors are positive or negative; the other is bounding-
[0087] Box regression: Correct the anchors to get more accurate proposals. Therefore, the RPN network is equivalent to making some predictions in advance, that is, judging whether there is a target and correcting the anchors to make the box more accurate.
[0088] 3) RoL Pooling (i.e., region of interest pooling) is used to collect proposals generated by RPN and extract them from the feature maps in 1) to generate proposal featuremaps (candidate box feature maps) and send them to the subsequent fully connected layer for classification and regression.
[0089] 4) Classification and Regression (using fully connected layers), use Proposals feature maps to calculate the specific category, and then do a bounding box regression to obtain the final precise position of the detection box.
[0090] In addition, the Fast RCNN network has two output layers of the same level: the cls_score layer and the bbox_pred layer, both of which are fully connected layers. The cls_score layer is used for classification, outputting a k+1-dimensional array p, which represents the probability of belonging to the k class and the background. It outputs a discrete probability distribution for each RoI (Region of Interesting), and p is calculated by the k+1 class fully connected layer using softmax. The bbox_pred layer is used to adjust the position of the candidate region and output the displacement of the bounding box regression. F.cross_entropy is the cross entropy loss function, which is generally used to calculate the loss after the fully connected layer; the loss function of the bounding box regression uses the Smooth L1 Loss function.
[0091] Further, after the establishment of the above image detection model is completed, the constructed image detection model can be trained using the pre-constructed training data set. The model training method is consistent with the conventional Faster-RCNN training method and will not be described in detail here.
[0092] Afterwards, the trained image detection model can be used to perform object detection on the image to be detected.
[0093] The image detection method based on Faster-RCNN of the present invention adopts a network model that integrates CNN and Transformer for feature extraction. Transformer can capture long-distance dependencies to extract global information, can retain global and local features to the greatest extent, can better extract local and global information of the image, and can better extract the global features of the image without increasing the size of the CNN convolution kernel, thereby improving the recognition rate.
[0094] Example 2
[0095] This embodiment provides an image detection device based on Faster-RCNN, including:
[0096] A feature extraction network, wherein the feature extraction network uses a network structure that integrates a Transformer module and a CNN network to extract features from an input image and generate a feature map;
[0097] Region proposal network, used to generate candidate boxes for locating objects;
[0098] The region of interest pooling layer is used to collect the candidate boxes generated by the region candidate module and extract the feature map from the feature extraction module to generate a candidate box feature map;
[0099] The fully connected layer is used to calculate the specific category using the candidate box feature map and obtain the final precise position of the detection box through bounding box regression.
[0100] Optionally, the feature extraction network includes: a 3*3 one-dimensional convolution layer, a 64*64 one-dimensional convolution layer, a 2*2 maximum pooling layer, a fully connected layer, a first normalization layer, a multi-head attention mechanism layer, a second normalization layer and a multi-layer perceptron layer connected in sequence;
[0101] Among them, a 3*3 one-dimensional convolution layer, a 64*64 one-dimensional convolution layer, a 2*2 maximum pooling layer and a fully connected layer constitute the CNN network, and the 3*3 one-dimensional convolution and the 64*64 one-dimensional convolution both use the ReLU activation function; the first normalization layer, the multi-head attention mechanism layer, the second normalization layer and the multi-layer perceptron layer constitute the Transformer module.
[0102] In this embodiment, the region candidate network is used to generate a candidate box for accurately locating an object using a kmeans algorithm.
[0103] The regional candidate networks include:
[0104] An anchor box generation module, used to generate an anchor box;
[0105] The anchor box extraction module is used to classify the generated anchor boxes using a softmax classifier and extract the anchor boxes with targets;
[0106] The bounding box regression module regresses the offset between the anchor box with the target and the annotated position of the real target to correct the anchor box, improve the accuracy of locating the object and output the candidate box.
[0107] The image detection device of this embodiment is Figure 2 The image detection model based on Faster-RCNN is shown. For the description of the relevant functions of the model, please refer to Example 1.
[0108] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.
Claims
1. An image detection method based on Faster-RCNN, characterized in that: include: Build an image detection model based on Faster-RCNN, including: Construct a feature extraction network that integrates the Transformer module and the CNN network, wherein the feature extraction network is used to extract features from the input image and generate a feature map; Constructing a region candidate network, wherein the region candidate network is used to generate a candidate box for locating an object; Constructing an area of interest pooling layer and a fully connected layer, wherein the area of interest pooling layer is used to collect candidate boxes generated by the area candidate network, and extract feature maps from the feature extraction network to generate candidate box feature maps, and the fully connected layer is used to calculate specific categories using the candidate box feature maps, and obtain the final precise position of the detection box through bounding box regression; Using a pre-constructed training data set to train the constructed image detection model; The trained image detection model is used to perform target detection on the image to be detected.
2. The image detection method based on Faster-RCNN according to claim 1, characterized in that: The feature extraction network includes: a 3*3 one-dimensional convolution layer, a 64*64 one-dimensional convolution layer, a 2*2 maximum pooling layer, a fully connected layer, a first normalization layer, a multi-head attention mechanism layer, a second normalization layer and a multi-layer perceptron layer connected in sequence; Among them, a 3*3 one-dimensional convolution layer, a 64*64 one-dimensional convolution layer, a 2*2 maximum pooling layer and a fully connected layer constitute the CNN network; the first normalization layer, the multi-head attention mechanism layer, the second normalization layer and the multi-layer perceptron layer constitute the Transformer module.
3. The image detection method based on Faster-RCNN according to claim 2, characterized in that: Both 3*3 one-dimensional convolution and 64*64 one-dimensional convolution use the ReLU activation function.
4. The image detection method based on Faster-RCNN according to claim 1, characterized in that: The region proposal network is used to generate candidate boxes for accurately locating objects using the kmeans algorithm.
5. The image detection method based on Faster-RCNN according to claim 4, characterized in that: The regional candidate networks include: An anchor box generation module, used to generate an anchor box; The anchor box extraction module is used to classify the generated anchor boxes using a softmax classifier and extract the anchor boxes with targets; The bounding box regression module regresses the offset between the anchor box with the target and the annotated position of the real target to correct the anchor box, improve the accuracy of locating the object and output the candidate box.
6. An image detection device based on Faster-RCNN, characterized in that: include: A feature extraction network, wherein the feature extraction network uses a network structure that integrates a Transformer module and a CNN network to extract features from an input image and generate a feature map; Region proposal network, used to generate candidate boxes for locating objects; The region of interest pooling layer is used to collect the candidate boxes generated by the region candidate module and extract the feature map from the feature extraction module to generate a candidate box feature map; The fully connected layer is used to calculate the specific category using the candidate box feature map and obtain the final precise position of the detection box through bounding box regression.
7. The image detection device based on Faster-RCNN according to claim 1, characterized in that: The feature extraction network includes: a 3*3 one-dimensional convolution layer, a 64*64 one-dimensional convolution layer, a 2*2 maximum pooling layer, a fully connected layer, a first normalization layer, a multi-head attention mechanism layer, a second normalization layer and a multi-layer perceptron layer connected in sequence; Among them, a 3*3 one-dimensional convolution layer, a 64*64 one-dimensional convolution layer, a 2*2 maximum pooling layer and a fully connected layer constitute the CNN network; the first normalization layer, the multi-head attention mechanism layer, the second normalization layer and the multi-layer perceptron layer constitute the Transformer module.
8. The image detection device based on Faster-RCNN according to claim 7, characterized in that: Both 3*3 one-dimensional convolution and 64*64 one-dimensional convolution use the ReLU activation function.
9. The image detection device based on Faster-RCNN according to claim 6, characterized in that: The region proposal network is used to generate candidate boxes for accurately locating objects using the kmeans algorithm.
10. The image detection device based on Faster-RCNN according to claim 9, characterized in that: The regional candidate networks include: An anchor box generation module, used to generate an anchor box; The anchor box extraction module is used to classify the generated anchor boxes using a softmax classifier and extract the anchor boxes with targets; The bounding box regression module regresses the offset between the anchor box with the target and the annotated position of the real target to correct the anchor box, improve the accuracy of locating the object and output the candidate box.
Citation Information
Cited By
Shadow operation target cloud layer identification method, system and device and medium
CN120339806A
