GCNN cross-task mode classification method based on transfer learning

Abstract

The invention relates to the technical field of growth convolutional neural network related algorithms, and discloses a transfer learning-based GCNN cross-task mode classification method, which comprises the following steps of pre-training an intermediate layer of a GCNN on a source task set, migrating parameters to a target task set, and removing an original FC6 layer. A solution is provided for the problem, weight parameters pre-trained on a source task set are migrated to a target task set, an adaptation layer is added, and training is carried out; cutting the target task set by using a sliding window, screening and marking the category of patches according to a frame overlapping rate, deleting patches containing a plurality of objects, and setting other patches which do not meet conditions as background categories; according to the transfer learning, model parameters in a source task set can be transferred to target tasks of a small sample set, the problem that the sample scale of a training set is too small is solved, the overfitting phenomenon of network training is relieved, and meanwhile the learning efficiency of a cross-task classification model can be improved.

Description

Technical Field

[0001] The present invention relates to growable convolutional neural network related algorithms, and in particular to a GCNN cross-task pattern classification method based on transfer learning. Background Art

[0002] As the difficulty of visual recognition tasks gradually increases, building a larger convolutional neural network (CNN) can achieve better recognition results. The results of the famous Large Scale Visual Recognition Challenge (ILSVRC) show that the improvement in network classification accuracy is largely due to the increase in network size. However, when the training samples are limited, although excessive neurons can easily fit the training data, they cannot reflect the true distribution of the test samples, resulting in an overfitting problem. Many scholars have proposed different solutions to this problem.

[0003] Hidden units with local receptive fields are defined through convolution operations, and the local spatial topological structure information of the image is integrated into the network with downsampling, which alleviates the impact of overfitting to a certain extent. However, the problem of too many network parameters is still not solved. Through pre-learning, a series of related restricted Boltzmann machines (RBMs) are used to mine useful information from training images to obtain a network structure with better initial weights. However, the additional time required to initialize the network layer by layer will increase with the increase in the number of layers, and the image information obtained through unsupervised learning may also have a negative impact. Therefore, a Dropout strategy is proposed. By randomly ignoring the weights of some hidden nodes in the network during model training, the joint action between some feature detectors is suppressed. Since the network models corresponding to the samples input into the network each time are different and share weights with each other, this greatly improves the stability and generalization ability of the network.

[0004] Existing studies have found that there is a certain relationship between the correlation between hidden layer activation units and overfitting, that is, when the correlation between hidden layer activation units decreases, the overfitting phenomenon of the network will be relatively alleviated. Another solution to the overfitting problem is to start with the training data set. Deep convolutional neural network (DCNN) can produce better results than traditional shallow learning methods, but as a large-capacity classifier with a huge number of parameters, it requires a large amount of training data.

[0005] For example, in the texture classification task, using a small number of low-resolution images from the Brodatz texture dataset, although the results were improved compared to other methods, they were not significant. The operators used the above model in the forest species classification task and achieved good results. The task used a training set with up to 3,000 images and each image was 3264x2448 in size. It can be seen that having a large training set is an important factor in training a convolutional neural network with excellent performance. However, obtaining labeled training sets is expensive or difficult, which inevitably causes the "data hunger" problem of convolutional neural networks.

[0006] The most direct solution to this problem is to use existing data to expand the training sample set. For example, operators use affine and elastic transformations such as rotation, scaling, and horizontal cropping to expand the MNIST handwriting dataset, thereby reducing the sensitivity of neurons to intra-class diversity and improving classification performance. In recent years, open labeled image datasets have become increasingly abundant, and the problem of "data hunger" has been alleviated. Operators first train the network on large datasets such as ImageNet and Places, and then use transfer learning to expand it to new recognition tasks. This can not only transfer the model parameters obtained from the source task training of the large-scale training set to the target task of the small sample set, solving the problem of the small sample size of the training set and alleviating the overfitting phenomenon to a certain extent, but also improve the learning efficiency of the cross-task classification model.

[0007] However, there are significant statistical differences between images in different datasets; for example, in datasets such as Caltech256 and ImageNet, objects are usually in the middle of the image, while in datasets such as Pascal VOC and LabelMe, objects are generally embedded in the scene. Therefore, the neural network trained in the source task may not be suitable for the target task set; when training and testing across task datasets, different scene environments, backgrounds (negative classes), and label biases (i.e., the object labels in the source task set are different from those in the target task set) will inevitably affect the recognition performance.

[0008] In practical applications, there are mainly the following problems:

[0009] (1) Additional time overhead: The additional time overhead required to initialize the network layer by layer increases significantly with the number of layers, which poses a challenge to the training efficiency of deep neural networks.

[0010] (2) Dataset statistical differences: The image statistics between different datasets are significantly different, which limits the applicability of the model parameters trained in the source task to the target task set, which affects the effect of cross-task pattern classification to a certain extent.

[0011] (3) Overfitting problem: When the training samples are limited, the network is prone to overfitting. Although overfitting can be alleviated to a certain extent through convolution operations and pre-learning strategies, these methods still have limitations, such as too many network parameters and the negative impact of unsupervised learning. Summary of the invention

[0012] The present invention aims to solve the problem of "data hunger" of GCNN by pre-training the intermediate layer of GCNN on the source task set, migrating the parameters to the target task set, removing the original FC6 layer, adding adaptation layers FCa and FCb, training only these two layers, using the output of the FC5 layer as the input of FCa, and processing through the adaptation layers FCa and FCb to achieve the classification of the target task set and compensate for the statistical differences and label bias between the source task set and the target task set; using a sliding window to crop the image of the target task set to generate multiple patches, screening and marking the categories of the patches according to the border overlap rate, deleting patches containing multiple objects, setting other patches that do not meet the conditions as background categories, replacing the last convolutional layer (C2) and the fully connected layer with a spatial pyramid pooling SPP layer, calculating the confidence that the image belongs to a certain category, and integrating the classification results of each patch to achieve image classification.

[0013] The present invention provides the following technical solution: a GCNN cross-task pattern classification method based on transfer learning, comprising the following steps:

[0014] Step 1: Pre-train the intermediate layers of GCNN on the source task set, transfer the parameters to the target task set, remove the original FC6 layer, add adaptation layers FCa and FCb, train only these two layers, use the output of the FC5 layer as the input of FCa, and process it through the adaptation layers FCa and FCb to achieve the classification of the target task set and compensate for the statistical differences and label bias between the source task set and the target task set;

[0015] Step 2: Use a sliding window to crop the image of the target task set to generate multiple patches, filter and mark the categories of the patches according to the border overlap rate, delete the patches containing multiple objects, and set other patches that do not meet the conditions as the background category; replace the last convolutional layer (C2) and the fully connected layer with a spatial pyramid pooling SPP layer, calculate the confidence that the image belongs to a certain category, and integrate the classification results of each patch to achieve image classification.

[0016] Preferably, the specific implementation steps of step 1 are:

[0017] 1): Pre-training and migration: The internal layers of GCNN (C1-FC5) are used as a general mid-level image feature extractor, which is first pre-trained on a large labeled dataset (called the source task set, ImageNet), and then the trained parameters are migrated to the target task set with limited training samples for reuse (such as Figure 1 As shown), the classification of the target task set is realized:

[0018] For the source task set, the outputs of the fully connected layers FC5 and FC6 (classification layers) of GCNN are Y5 and Y6 respectively, where Y k represents the output of the kth layer, W k , B k Represents the weight and bias parameters of the kth layer:

[0019] Y b -σ(W b Y4+B b )

[0020] Y6=φ(W6Y6+B6)

[0021] σ(X)[i] and φ(X)[i] are ReLU and softmax nonlinear activation functions respectively:

[0022] σ(X)[i]=max(0,X[i])

[0023]

[0024] 2): Adaptation and classification: In order to compensate for the statistical differences and label biases between the source and target task sets, the fully connected layer FC6 of the original GCNN can be removed in the target task, and two new adaptation layers FCa and FCb can be added (such as Figure 2 As shown in the figure, in the target task, only the two adaptation layers are trained for the labeled target task set, where the output Y5 of FC5 is used as the input of FCa, and the outputs of FCa and FCb are Y a and Y b , W a , W b , B a , B b are the parameters that need to be learned:

[0025] Y a =σ(W a Y5+B a )

[0026] Y b =φ(W b Y a +B b ).

[0027] Preferably, the specific implementation steps of step 2 are:

[0028] 1) Data preprocessing: To address the data capture bias problem in cross-task transfer learning, that is, the source task dataset is relatively simple, each image contains only a single object, while the images in the target task dataset may contain complex scenes such as multiple objects of different scales. A sliding window can be used to crop the labeled images in the target task set. Each image gets multiple patches, which may contain one object, multiple objects, or background. In order to label these patches in the training set, the bounding box B of each patch is compared with the bounding box B of the labeled object Po in the image. u Calculate the overlap ratio if the following three conditions are met:

[0029] (a)|B∩B o |≥η1|B|

[0030] (b)|B∩B o | ≥η2|B o |

[0031] (c) The patch does not overlap with multiple objects in the image;

[0032] The category of the patch is set to the category of Po, where |·| represents the area of ​​the border, ∩ represents the calculation of the overlapping part of the two targets, 0<η1, η2<1, and for a patch, if it contains more than one object, the patch is directly deleted, and other patches that do not meet the above conditions are set to the background category;

[0033] 2): SPP layer replacement: Since the scales of these patches are different, in order to be able to input into GCNN so that it can accept inputs of different sizes but produce outputs of the same size, the sampling layer between the last convolutional layer and the fully connected layer is replaced with a spatial pyramid pooling (SPP) layer (such as Figure 2 As shown), when classifying, the following formula is used to calculate whether the image belongs to category C i Confidence of:

[0034]

[0035] Compared with the existing GCNN cross-task pattern classification method based on transfer learning, the present invention has the following advantages:

[0036] Beneficial effects:

[0037] The present invention proposes a solution to the above problems, by migrating the weight parameters pre-trained on the source task set to the target task set, adding an adaptation layer, and performing training; using a sliding window to crop the target task set, screening and marking the category of patches according to the border overlap rate, deleting patches containing multiple objects, and setting other patches that do not meet the conditions as background categories; transfer learning can not only migrate the model parameters in the source task set to the target task in the small sample set, solve the problem of too small sample size in the training set, alleviate the overfitting phenomenon of network training, but also improve the learning efficiency of the cross-task classification model. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a GCNN cross-task pattern classification method based on transfer learning;

[0039] Figure 2 Schematic diagram of the GCNN network structure with SPP layer and adaptation layer added. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] See also Figure 1 and Figure 2 ,The GCNN cross-task mode classification method based on transfer learning includes the following steps:

[0042] Step 1: Pre-train the intermediate layers of GCNN on the source task set, transfer the parameters to the target task set, remove the original FC6 layer, add adaptation layers FCa and FCb, train only these two layers, use the output of the FC5 layer as the input of FCa, and process it through the adaptation layers FCa and FCb to achieve the classification of the target task set and compensate for the statistical differences and label bias between the source task set and the target task set;

[0043] Step 2: Use a sliding window to crop the image of the target task set to generate multiple patches, filter and mark the categories of the patches according to the border overlap rate, delete the patches containing multiple objects, and set other patches that do not meet the conditions as the background category; replace the last convolution layer (C2) and the fully connected layer with a spatial pyramid pooling SPP layer, calculate the confidence that the image belongs to a certain category, and integrate the classification results of each patch to achieve image classification;

[0044] The specific implementation steps of step one are:

[0045] 1. Pre-training and migration: The internal layers of GCNN (layers C1-FC5) are used as a general mid-level image feature extractor, which is first pre-trained on a large labeled dataset (called the source task set, ImageNet), and then the trained parameters are migrated to the target task set with limited training samples for reuse (such as Figure 1 As shown), the classification of the target task set is realized.

[0046] For the source task set, the outputs of the fully connected layers FC5 and FC6 (classification layers) of GCNN are Y5 and Y6 respectively, where Y k represents the output of the kth layer, W k , B k represents the weight and bias parameters of the kth layer,

[0047] Y5=σ(W6Y4+B6)

[0048] Y6=φ(W6Y5+B6)

[0049] σ(X)[i] and φ(X)[i] are ReLU and softmax nonlinear activation functions respectively.

[0050] σ(X)[i]=max(0,X[i])

[0051]

[0052] 2. Adaptation and classification: In order to compensate for the statistical differences and label biases between the source and target task sets, the fully connected layer FC6 of the original GCNN can be removed in the target task, and two new adaptation layers FCa and FCb can be added (such as Figure 2 As shown in the figure, in the target task, only the two adaptation layers are trained for the labeled target task set, where the output Y5 of FC5 is used as the input of FCa, and the outputs of FCa and FCb are Y a and Y b , W a , W b , B a , B b are the parameters that need to be learned.

[0053] Y a =σ(W a Y6+B a )

[0054] Y b =φ(W b Y a +B b ).

[0055] The specific implementation steps of step 2 are:

[0056] 1. Data preprocessing: To address the data capture bias problem in cross-task transfer learning, that is, the source task dataset is relatively simple, each image contains only a single object, while the images in the target task dataset may contain complex scenes such as multiple objects of different scales, a sliding window can be used to crop the images in the labeled target task set, and each image will get multiple patches, which may contain one object, multiple objects or background. In order to label these patches in the training set, the bounding box B of each patch is compared with the bounding box B of the labeled object Po in the image. o Calculate the overlap ratio if the following three conditions are met:

[0057] (a)|B∩B o |≥η1|B|

[0058] (b)|B∩B o |≥η2|B o |

[0059] (c) The patch does not overlap with multiple objects in the image;

[0060] The category of the patch is set to the category of Po, where |·| represents the area of ​​the border, ∩ represents the calculation of the overlapping part of the two targets, 0<η1, η2<1. For a patch, if it contains more than one object, the patch is directly deleted, and other patches that do not meet the above conditions are set to the background category:

[0061] Second: SPP layer replacement: Since the scales of these patches are different, in order to be able to input into GCNN so that it can accept inputs of different sizes but produce outputs of the same size, the sampling layer between the last convolutional layer and the fully connected layer is replaced with a spatial pyramid pooling (SPP) layer (such as Figure 2 As shown), when classifying, the following formula is used to calculate whether the image belongs to category c i Confidence of:

[0062]

[0063] Where y(C i | P m ) indicates patchP m Belongs to category c i The probability of , which is the output of softmax, M is the number of patches extracted from the image, and the parameter l ≥ 1 is used to weaken the contribution of patches with smaller classification confidence.

[0064] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A GCNN cross-task pattern classification method based on transfer learning, characterized in that: The following steps are involved: Step 1: Pre-train the intermediate layers of GCNN on the source task set, transfer the parameters to the target task set, remove the original FC6 layer, add adaptation layers FCa and FCb, and only train these two layers. Use the output of the FC5 layer as the input of FCa, and process it through the adaptation layers FCa and FCb to achieve the classification of the target task set and compensate for the statistical differences and label bias between the source task set and the target task set. Step 2: Use a sliding window to crop the image of the target task set to generate multiple patches, filter and mark the categories of the patches according to the border overlap rate, delete the patches containing multiple objects, and set other patches that do not meet the conditions as the background category; replace the last convolutional layer (C2) and the fully connected layer with a spatial pyramid pooling SPP layer, calculate the confidence that the image belongs to a certain category, and integrate the classification results of each patch to achieve image classification.

2. The GCNN cross-task pattern classification method based on transfer learning according to claim 1 is characterized in that: The specific implementation steps of step 1 are: 1): Pre-training and migration: The internal layers of GCNN (C1-FC5) act as a general mid-level image feature extractor, which is first pre-trained on a large labeled dataset (called the source task set, ImageNet), and then the trained parameters are migrated to the target task set with limited training samples for reuse (as shown in Figure 1) to achieve classification of the target task set: For the source task set, the outputs of the fully connected layers FC5 and FC6 (classification layers) of GCNN are Y5 and Y6 respectively, where Y k represents the output of the kth layer, W k , B k Represents the weight and bias parameters of the kth layer: Y6=σ(W6Y4+B6) Y6=φ(W6Y4+B6) σ(X)[t] and φ(X)[t] are ReLU and softmax nonlinear activation functions respectively: σ(X)[i]=max(0,X[i]) 2) Adaptation and classification: In order to compensate for the statistical differences and label bias between the source and target task sets, the fully connected layer FC6 of the original GCNN can be removed in the target task, and two new adaptation layers FCa and FCb can be added (as shown in Figure 2). In the target task, these two adaptation layers are only trained for the labeled target task set, where the output Y0 of FC5 is used as the input of FCa, and the outputs of FCa and FCb are Y a and Y b , W a , W b , B a , B b are the parameters that need to be learned: Y a =σ(W n Y b +B n ) Y b =φ(W b Y a +B b )。 3. The GCNN cross-task pattern classification method based on transfer learning according to claim 1, characterized in that: The specific implementation steps of step 2 are: 1) Data preprocessing: To address the data capture bias problem in cross-task transfer learning, that is, the source task dataset is relatively simple, each image contains only a single object, while the images in the target task dataset may contain complex scenes such as multiple objects of different scales. A sliding window can be used to crop the labeled images in the target task set. Each image gets multiple patches, which may contain one object, multiple objects, or background. In order to label these patches in the training set, the bounding box B of each patch is compared with the bounding box B of the labeled object Po in the image. o Calculate the overlap ratio if the following three conditions are met: (a)|B∩B v |≥η1|B| (b)|B∩B o |≥η2|B o | (c) The patch does not overlap with multiple objects in the image; The category of the patch is set to the category of Po, where |·| represents the area of ​​the border, ∩ represents the calculation of the overlapping part of the two targets, 0<η1, η2<1, and for a patch, if it contains more than one object, the patch is directly deleted, and other patches that do not meet the above conditions are set to the background category; 2): SPP layer replacement: Since the scales of these patches are different, in order to be able to input into GCNN so that it can accept inputs of different sizes but produce outputs of the same size, the sampling layer between the last convolutional layer and the fully connected layer is replaced with a spatial pyramid pooling (SPP) layer (as shown in Figure 2). When classifying, the following formula is used to calculate whether the image belongs to category C i Confidence of: Where y(C i |P m ) indicates patch P m Belongs to category c i The probability of , which is the output of softmax, M is the number of patches extracted from the image, and the parameter l ≥ 1 is used to weaken the contribution of patches with smaller classification confidence.

Images