A shape prior based object detection method

Abstract

The application discloses a target detection method based on shape prior, and specifically comprises a target detection method for a large-scale data set and a target detection method for a small sample data set. The application firstly constructs a shape data set, then designs a target detection model, and pretrains the model through the shape data set, and finally fine-tunes the weight of the model through a target task data set (usually a large-scale data set), so that the target detection of the large-scale data set is realized. On this basis, the application designs a small sample target detection model, trains part of the structure in the model through the shape data set and freezes, and then trains the remaining structure in the model through a target task data set (usually a small sample data set), so that the target detection of the small sample data set is finally realized.

Description

TECHNICAL FIELD

[0001] The present application relates to a target detection method, in particular to a target detection method based on shape prior, and belongs to the field of computer vision. BACKGROUND

[0002] Convolutional neural networks (CNNs) have shown impressive performance in a wide range of visual tasks, but compared to the human visual system, they show surprisingly fragile performance on many tasks. For example, image CNNs struggle on "domain transfer" tasks, where the conditions of training do not match the conditions of testing. Furthermore, small corruptions such as noise, blur, fog, and perturbations that do not affect human perception can affect the performance of image CNNs. In particular, when these small changes are adversarial, i.e., they are targeted at the architecture of the CNN to get the maximum effect, the performance of the CNN can be severely affected, such as: adversarial attacks.

[0003] The original intention of designing a deep neural network is to replace human decision-making, and the performance of CNN is completely opposite to the shape prior of human vision relative to the global and local shape of the picture: in the picture, the convolutional neural network only captures a small amount of local shape and a large amount of texture information, while humans tend to capture global shape information. Relative to texture, shape is a constant property of an object, and the shape of an object can remain unchanged under picture corruption or perturbation. It is generally believed that the strong shape preference of the human visual system is an important reason for its extremely strong robustness, and the fundamental reason for the weakness of CNN is the lack of shape description. SUMMARY

[0004] The present application is inspired by the behavior of human infants recognizing objects through shape, and the main content of the application is to propose a target detection method based on shape prior, which can be applied to both large-scale data sets and small-sample data sets.

[0005] The technical scheme adopted by the present application is a target detection method based on shape prior, and the implementation process of the method is as follows:

[0006] Step 1: Construct the original shape data set. Use a custom script to generate several pictures, each of which contains four types of basic shapes: rectangles, circles, triangles, and ellipses, and use the four types of basic shapes in the picture as input samples of the original shape data set; the class and the coordinates of the circumscribed rectangle of the basic shape contained in the picture are used as the expected output label.

[0007] Step 2: Constructing the stylized shape dataset. The pictures in the original shape dataset are self-stylized to obtain the input samples of the stylized shape dataset, and the expected output uses the expected output of the corresponding picture in the original shape dataset.

[0008] Step 3: Constructing the shape dataset. The original shape dataset and the stylized shape dataset are combined to obtain the shape dataset. The target detection of the shape dataset is realized based on the target detection of the large-scale dataset and the target detection of the small sample dataset.

[0009] The target detection of the large-scale dataset includes steps 4 and 5:

[0010] Step 4: Designing the target detection model. Based on the RPN model, the channel number of the 1x1 convolution kernel in the RPN model is modified. The channel number of the convolution kernel is 18, i.e. 9 anchor boxes and 2 scores per point. The channel number of the convolution kernel is modified to 45, i.e. 9 anchor boxes and 5 scores per point. The modified RPN model is the target detection model.

[0011] Step 5: Training the target detection model. The shape dataset obtained in step 3 is used to train the target detection model in step 4 for the first time to obtain the first group of model weights. The target detection model is trained for the second time through the target task dataset, so that the first group of model weights can be fine-tuned according to the target task dataset to obtain a group of final model weights for the target detection task of the large-scale dataset. The target detection of the large-scale dataset can be realized through the target detection model designed in step 4 and the final model weights obtained by training in step 5.

[0012] The target detection of the small sample dataset includes step 6:

[0013] Step 6: Designing and training the small sample target detection model. The small sample target detection model is based on the Faster R-CNN model, and the RPN model in the Faster R-CNN model is modified to the ShapeRCN shape detector, and a shape prior branch is added. The modified Faster R-CNN model is named shape prior Faster R-CNN model, and the backbone network and ShapeRCN shape detector in the shape prior Faster R-CNN model are trained through the shape dataset in step 3 and frozen. Then, the other parts of the Faster R-CNN model except the backbone network and the ShapeRCN shape detector are trained through the target task dataset, so as to realize the target detection of the small sample dataset.

[0014] Further, the specific implementation of step 6 includes the following:

[0015] Step 61) Design and train the ShapeRCN shape detector. Based on the Faster R-CNN model, modify the channel number of the 1x1 convolution kernel in the RPN model in the Faster R-CNN model to 45, and the modified RPN model is called ShapeRCN shape detector. Train the backbone network and ShapeRCN shape detector in the modified Faster R-CNN network through the shape data set, obtain the weights of the backbone network and ShapeRCN shape detector and fix them, so that the backbone network and ShapeRCN shape detector do not participate in the subsequent training process, that is, freeze the backbone network and ShapeRCN shape detector.

[0016] Step 62) Introduce shape prior branch. On the basis of the Faster R-CNN model, increase the shape prior branch structure. The shape prior branch structure first obtains the shape category features output by the ShapeRCN shape detector, and each shape category feature is a ROI region feature map of size D roi ×D roi , and the number of shape category features is C shape , C shape ∈{0,1,2,3}, the ROI region feature map is M shape , M shape ∈{M0,M1,…,M Cshape}, wherein M0,M1,…,M Cshape respectively represent the ROI region feature maps corresponding to the first feature, the second feature, …, and the C shape th feature. M shape will be used as shape category prior information for subsequent shape category embedding.

[0017] Step 63) Position encoding and shape embedding. Each input picture contains N targets, and there may be targets of the same class in the N targets. Then the ShapeRCN shape detector will output N proposal boxes and N shape categories, and each proposal box corresponds to a shape category. The shape feature map M shape corresponding to each shape category is input into the fixed position encoding module. In the position encoding module, different frequencies of sine and cosine functions are used to encode the shape category, as shown in equations (1) and (2).

[0018]

[0019] Wherein, i is the index of the proposal box, satisfying i∈{0,1,…,N proposal}, N proposal is the number of proposal boxes output by the ShapeRCN shape detector, and D modelis the output feature map dimension of the feature extraction network backbone in the Faster R-CNN model, and is also the input dimension of the RCNN shared fully connected layer. Each shape embedding class corresponds to a sine wave, and the wavelength of the sine wave forms a geometric progression from 2π to 1000 times 2π. A total of N feature vectors with dimensions D model ×D roi ×D roi are obtained. The shape prior feature dimension D model output by the shape embedding module is the same as the input dimension of the shared feature layer, and is added and input into the subsequent network.

[0020] Step 64) Shape prior Faster R-CNN model training. According to step 61), the backbone network and the ShapeRCN shape detector in the shape prior Faster R-CNN model are trained through the shape data set, and the backbone network and the ShapeRCN shape detector do not participate in subsequent training, that is, the backbone network and the ShapeRCN shape detector in the shape prior Faster R-CNN model are frozen. The other parts of the shape prior Faster R-CNN model except the backbone network and the ShapeRCN shape detector are trained through the target task data set, and after the training is completed, the target shape prior Faster R-CNN model suitable for the small sample data set is obtained.

[0021] Compared with the prior art, the shape prior module is introduced into the conventional target detection model, and the accuracy of target detection is effectively improved. In theory, the feature extraction structure (such as a convolutional layer) in the target detection model can be considered as filtering the sample to obtain different feature information. For a relatively shallow feature extraction structure, the extracted feature information is relatively basic, such as point features, line features, and surface features. For a relatively deep feature extraction structure, the extracted feature information is closer to the semantic information in the target task data set.

[0022] Generally, in the training phase of the network model, the shallow features and the deep features are trained through the target data set, but the semantic information of the target data set is usually rich, and the texture information is also very complex. For the deep feature extraction structure, since the semantic information in the data set needs to be extracted, the real target data set is the best data source. However, for the shallow feature extraction structure, the overly complex texture information in the real target data set will seriously affect the extraction effect and efficiency of the shallow features, and the shallow feature extraction result is the data basis for deep feature extraction. Therefore, if appropriate shallow features cannot be extracted, the extraction effect of deep features will also be poor, and in the form of the final output result, the target in the data set cannot be effectively detected.

[0023] Based on the above theoretical analysis, the application proposes the concept of shape prior, which can be considered as a result of fast, efficient and accurate training of shallow features in a deep network model in theory. Since shallow features are mainly point features, line features, surface features, etc. (such as color, texture, edge, corner), the model trained by constructing a shape dataset can be directly used as a shallow feature extraction structure in a large model by migrating the trained model to a traditional large model. And the self-styling processing of the shape dataset essentially improves the diversity of samples to ensure that the shallow feature extraction structure has sufficient generalization ability. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 Shape dataset generation schematic diagram.

[0025] Figure 2 Shape pre-training schematic diagram.

[0026] Figure 3 Small sample target detection network based on shape prior.

[0027] Figure 4 Shape category embedding schematic diagram. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical scheme and advantages of the application clearer, the application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the application and not to limit the application.

[0029] REFERENCE Figures 1-4 , examples:

[0030] A target detection method based on shape prior, the target detection method based on shape prior comprises the following steps:

[0031] S1: Construct an original shape dataset.

[0032] First, generate input sample pictures of the original shape dataset. Use a custom Python script and OpenCV library to generate several pictures, each of which contains four basic shapes: rectangles, circles, triangles and ellipses. The picture resolution is 1333x800. In order to simulate the diversity of shapes in the real world, the number, position, color and rotation angle of each shape in each picture are controllable.

[0033] Secondly, the expected output label of the original shape dataset is generated. Since the number, position, color and rotation angle of each shape in each picture of the generated sample picture are determined by controllable input parameters, the class and bounding rectangle coordinates of the shape contained in each picture can be calculated according to these parameters, and the class and bounding rectangle coordinates of the shape in each picture are taken as the expected output label corresponding to the picture. Thus, the original shape dataset can be obtained.

[0034] S2: Constructing a stylized shape dataset.

[0035] The pictures in the original shape dataset are subjected to AdaIN style transfer self-stylization processing to enrich the texture information and background information of the pictures, but without changing the number, position, color and rotation angle of each shape in the pictures, as shown in Figure 1 The pictures after self-stylization processing are stored separately as input samples of the stylized shape dataset, and the expected output can still use the expected output of the corresponding picture in the original shape dataset, so that the stylized shape dataset can be obtained.

[0036] S3: Constructing a shape dataset.

[0037] The format and size of the pictures in the original shape dataset and the stylized shape dataset are the same, and the format of the label is also the same. The original shape dataset and the stylized shape dataset can be combined to obtain a shape dataset for subsequent experiments.

[0038] S4: Designing a target detection model.

[0039] The target detection model is mainly used for target detection tasks in large-scale datasets. The model is based on the RPN model, and the channel number of the 1x1 convolution kernel in the RPN model is modified. Since the RPN model is originally used for binary classification detection of targets, only whether the target belongs to the foreground or the background is judged, so the channel number of the convolution kernel is 18, that is, 9 anchor boxes and 2 scores for each point. However, the target type of the shape dataset constructed in the present application has a total of 5 types, that is, 4 shape targets and 1 background, so the channel number of the convolution kernel is modified to 45, that is, 9 anchor boxes and 5 scores for each point. The modified RPN model is the target detection model.

[0040] S5: Training the target detection model.

[0041] The target detection model is trained for the first time through the shape dataset to obtain a group of model weights; on this basis, the feature extraction layer in part of the model is frozen, and then the target detection model is trained and verified on various downstream tasks using the target task dataset (large-scale dataset), so that the weights of the feature extraction layer in the model that is not frozen can be fine-tuned according to the target task dataset, thereby realizing target detection on the large-scale dataset, as shown inFigure 2 The upper part of the block diagram is shown.

[0042] S6: Design and train a small sample target detection model.

[0043] The small sample target detection model is based on the Faster R-CNN model, modifies the convolution kernel of the RPN model in the Faster R-CNN model, and adds a shape prior branch structure, such as Figure 2 The lower part of the block diagram is shown. The modified model is called a shape prior Faster R-CNN model, which can improve the performance of the small sample detection task without increasing additional training parameters. The shape prior Faster R-CNN model and the original Faster R-CNN model are compared as shown in Figure 3 .

[0044] S6-1: Design and train a ShapeRCN shape detector.

[0045] Similar to the method in S4, based on the Faster R-CNN model, the number of channels of the 1x1 convolution kernel in the RPN model of the Faster R-CNN model is modified to 45. The modified RPN model is called ShapeRCN shape detector. The backbone network and ShapeRCN shape detector in the Faster R-CNN model are trained through a shape data set to obtain a set of weights of the backbone network and ShapeRCN. Then, the weights are fixed so that the backbone network and ShapeRCN do not participate in the training process in the subsequent steps, i.e., the backbone network and ShapeRCN shape detector are frozen.

[0046] When training the backbone network and ShapeRCN shape detector through the shape data set, the parameters are set as follows: the batch size is 128, the maximum number of iterations is 50, and the weights of the model are the weights corresponding to the 50th iteration.

[0047] S6-2: Introduce a shape prior branch.

[0048] On the basis of the Faster R-CNN, a shape prior branch structure is added. This structure first obtains the shape category features output by the ShapeRCN shape detector. Each category feature is a ROI region feature map of size D roi ×D roi , and the number of shape category features is C shape , C shape ∈{0,1,2,3}, the ROI region feature map is M shape , M shape ∈{M0,M1,…,M Cshape}, wherein M0, M1, …, M Cshape respectively represent the ROI region feature maps corresponding to the first type of feature, the second type of feature, …, and the C shape th type of feature. M shape The shape category prior information is used for subsequent shape category embedding. In the subsequent training and inference process, the feature map corresponding to each shape category does not change.

[0049] S6-3: Position encoding and shape embedding.

[0050] As Figure 4 shown, for each input image, the ShapeRCN shape detector also outputs N and N shape categories corresponding to the proposal boxes. Each shape category is mapped to the corresponding shape feature map M shape , which is input into the fixed position encoding module. In the position encoding module, different frequencies of sine and cosine functions are used to encode the shape category, as shown in equations (1) and (2).

[0051]

[0052] wherein M shape is defined in S6-2, i is the index of the proposal box, satisfying i∈{0,1,…,N proposal}, N proposal is the number of proposal boxes output by the ShapeRCN shape detector. D model is the output feature map dimension of the feature extraction network backbone in the Faster R-CNN model, which is also the input dimension of the RCNN shared fully connected layer. Each shape embedding category corresponds to a sine wave, and the wavelength of the sine wave forms a geometric progression from 2π to 1000 times 2π. Finally, N feature vectors with dimensions D model ×D roi ×D roi are obtained. The shape prior feature dimension D model output by the shape embedding module is the same as the input dimension of the shared feature layer, and is added and input into the subsequent network.

[0053] S6-4: Shape prior Faster R-CNN model training. According to S6-1, the backbone network and ShapeRCN shape detector in the shape prior Faster R-CNN model have been trained through the shape data set, so the backbone network and ShapeRCN will no longer participate in the subsequent training process, that is, the backbone network and ShapeRCN shape detector in the shape prior Faster R-CNN model are frozen. Finally, the parts other than the backbone network and ShapeRCN shape detector in the shape prior Faster R-CNN model are trained through the target task data set, and after the training is completed, the shape prior Faster R-CNN model suitable for small sample targets is obtained. When training the shape prior Faster R-CNN model through the target task data set, the training parameters are set as follows: the batch size is 32, and the weight of the model uses the weight corresponding to the model with the highest test set accuracy in all iteration processes.

[0054] The pseudo code of the small sample target detection model training process is as follows:

[0055]

Claims

1. A shape prior based object detection method, characterized in that: The implementation process of the method is as follows: Step 1: Constructing the original shape data set; using a custom script to generate several pictures, each of which contains four types of basic shapes: rectangles, circles, triangles and ellipses, and the four types of basic shapes in the picture are used as the input samples of the original shape data set; the category and circumscribed rectangle coordinates of the basic shape contained in the picture are used as the expected output label; Step 2: Constructing a stylized shape data set; performing self-stylization processing on the pictures in the original shape data set to obtain the input samples of the stylized shape data set, and the expected output uses the expected output of the corresponding picture in the original shape data set; Step 3: Constructing a shape data set; combining the original shape data set and the stylized shape data set, i.e. obtaining the shape data set; Target detection of the shape data set is realized based on target detection of a large-scale data set and target detection of a small sample data set; The target detection of the large-scale data set includes steps 4 and 5: Step 4: Designing a target detection model; based on the RPN model, modify the channel number of the 1x1 convolution kernel in the RPN model; the channel number of the convolution kernel is 18, i.e. 9 anchor boxes and 2 scores per point; modify the channel number of the convolution kernel to 45, i.e. 9 anchor boxes and 5 scores per point, and the modified RPN model is the target detection model; Step 5: Training the target detection model; the shape data set obtained in step 3 is used to train the target detection model in step 4 for the first time to obtain the first group of model weights; Through the second training of the target detection model by the target task data set, the first group of model weights can be fine-tuned according to the target task data set to obtain a group of final model weights for the large-scale data set target detection task, and the target detection of the large-scale data set can be realized through the target detection model designed in step 4 and the final model weights obtained by training in step 5; The target detection of the small sample data set includes step 6: Step 6: Design and train a small sample target detection model; the small sample target detection model is based on the Faster R-CNN model, the RPN model in the Faster R-CNN model is modified to the ShapeRCN shape detector, and a shape prior branch is added; the modified Faster R-CNN model is named as the shape prior Faster R-CNN model, and the backbone network and ShapeRCN shape detector in the shape prior Faster R-CNN model are trained through the shape data set in step 3 and frozen; then the other parts of the shape prior Faster R-CNN model except the backbone network and ShapeRCN shape detector are trained through the target task data set, so as to realize the target detection of the small sample data set.

2. The shape prior based object detection method of claim 1, wherein: The specific implementation of step 6 includes the following: Step 61) design and train the ShapeRCN shape detector; based on the Faster R-CNN model, the channel number of the 1x1 convolution kernel in the RPN model in the Faster R-CNN model is modified to 45, and the modified RPN model is called ShapeRCN shape detector; the backbone network and the ShapeRCN shape detector in the modified Faster R-CNN network are trained through the shape data set, and the weights of the backbone network and the ShapeRCN shape detector are obtained and fixed, so that the backbone network and the ShapeRCN shape detector do not participate in the subsequent training process, that is, the backbone network and the ShapeRCN shape detector are frozen; Step 62) introducing a shape prior branch; on the basis of the Faster R-CNN model, a shape prior branch structure is added; the shape prior branch structure first obtains shape category features output by a ShapeRCN shape detector, each shape category feature is an ROI region feature map of size D roi ×D roi , and the number of shape category features is C shape , C shape ∈{0, 1, 2, 3}, the ROI region feature map is M shape , M shape ∈{M0, M1,…, M Cshape}, wherein M0, M1,…, M Cshape represent the first type of feature, the second type of feature, and the C shape th type of feature corresponding ROI region feature map respectively; M shape will be used as shape category prior information for subsequent shape category embedding; Step 63) position coding and shape embedding; each input picture contains N targets, and there can be the same target in N targets, then the ShapeRCN shape detector outputs N proposal boxes and N shape categories, each proposal box corresponds to a shape category; the shape feature map M corresponding to each shape category is input into the fixed position coding module; in the position coding module, the shape category is coded using sine and cosine functions of different frequencies, as shown in formula (1) and formula (2). shape Input into the fixed position coding module; in the position coding module, the shape category is coded using sine and cosine functions of different frequencies, as shown in formula (1) and formula (2). wherein i is the index of the proposal box, satisfying i ∈ {0, 1, …, N proposal}, N proposal is the number of proposal boxes output by the ShapeRCN shape detector, D model is the output feature map dimension of the feature extraction network backbone in the Faster R-CNN model, which is also the input dimension of the RCNN shared fully connected layer; each shape embedding class corresponds to a sine wave, and the wavelength of the sine wave forms a geometric progression from 2π to 1000 times 2π; a total of N feature vectors with dimensions D model ×D roi ×D roi ; the shape prior feature dimension D model output by the shape embedding module is the same as the input dimension of the shared feature layer, and is added to the subsequent network. Step 64) shape prior Faster R-CNN model training; according to step 61), the backbone network and the ShapeRCN shape detector in the shape prior Faster R-CNN model are trained through the shape data set, and the backbone network and the ShapeRCN shape detector do not participate in the subsequent training, that is, the backbone network and the ShapeRCN shape detector in the shape prior Faster R-CNN model are frozen; finally, the parts other than the backbone network and the ShapeRCN shape detector in the shape prior Faster R-CNN model are trained through the target task data set, and the corresponding weights of each part are obtained, and at this time, the target shape prior Faster R-CNN model suitable for small sample data set can be obtained.

Images