A small sample target detection method based on feature space category orthogonalization

Abstract

The present invention discloses a small-sample target detection method based on feature space category orthogonalization, comprising: constructing a first detector based on the Faster-RCNN model and pre-training the first detector on a base class; constructing a second detector after connecting a newly added second category classifier to the end of the first detector, and fine-tuning the second detector on a new class, wherein the initial weight vector of the second category classifier is determined by a feature category orthogonalization module; and performing target detection on a newly input image using the trained second detector. The present invention adopts a category orthogonalization method, adding a new classifier layer to the last classifier layer of the first detector pre-trained on the base class, thereby achieving a transition when the second detector migrates from the base class to the new class, cleverly resolving the problems of inconsistent number of base categories and misaligned content in the prior art.

Description

Technical Field

[0001] The present invention relates to the field of target detection, and in particular to a small sample target detection method based on feature space category orthogonalization. Background Art

[0002] Object detection is an important and challenging computer vision task. Its primary goal is to locate and classify visual object instances of a specific category (such as humans, animals, or cars) in digital images. It is the foundation of many other vision tasks, such as instance segmentation, image captioning, and object tracking. In recent years, the rapid development and significant breakthroughs of deep learning have injected new life into object detection, propelling it to unprecedented research hotspots and finding widespread application in fields such as autonomous driving, video surveillance, and intelligent transportation.

[0003] Current detectors are mainly divided into one-stage and two-stage methods. The former directly predicts objects using simple convolutional neural networks, while the latter extracts proposals across different receptive fields and then performs refined predictions. Two-stage methods are generally more accurate than one-stage methods, but take slightly longer. However, these detectors all rely on a large amount of labeled data for training. Obtaining sufficient labeled data is usually labor-intensive, and they suffer from overfitting problems when faced with limited data, making them unable to generalize to new tasks with a small number of images. This severely limits the application of object detectors in current vision systems. In contrast, humans can observe new objects with limited examples. Therefore, few-shot object detection (FSOD) was proposed to bridge the gap between deep learning and human learning systems.

[0004] Mainstream methods for small-shot object detection fall into three categories: data augmentation-based, meta-learning-based, and transfer learning-based. Data augmentation-based methods alleviate data scarcity by expanding the data or extracting various generalizable features from limited data. Meta-learning-based methods intuitively and explicitly simulate small-shot scenarios involving both a query set and a support set, applying query and support branches to the two sets respectively. The most representative is Meta YOLO, based on YOLOv2, which extracts reweighted vectors from images in the Support Set and applies them as channel-level attention to detect objects in the Support Set. Meta R-CNN also extracts category prototypes from images in the Support Set and uses them as region-of-interest (RoI) features in the Query Set to detect objects. Transfer learning-based methods utilize fine-tuning or metric learning to transfer knowledge learned on a data-rich base class dataset to a data-scarce novel class dataset.

[0005] However, the existing technology has the following problems: when migrating a model pre-trained on a base class with a large amount of labeled data to a new class with sparse data, due to the inconsistent number of categories and misaligned content between the base class and the new class, the initial weights of the model's classifier need to be randomized during fine-tuning, resulting in a cold start problem for the classifier during fine-tuning, which greatly reduces the quality and convergence speed of the model. Summary of the Invention

[0006] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a small sample target detection method based on feature space class orthogonalization.

[0007] The object of the present invention is achieved through the following technical solutions:

[0008] A first aspect of the present invention provides a small sample target detection method based on feature space class orthogonalization, comprising the following steps:

[0009] Constructing a first detector based on the Faster-RCNN model and pre-training the first detector on the base class;

[0010] A second detector is constructed by connecting a newly added second-category classifier New Classifier to the end of the first detector, and fine-tuning the second detector on the new class. The initial weight vector of the second-category classifier New Classifier is determined by the feature category orthogonalization module.

[0011] Use the trained second detector to perform object detection on the newly input input image;

[0012] The feature category orthogonalization module includes:

[0013] Category orthogonalization unit: performs category orthogonal projection on the feature vector output by the first detector, mapping samples of each category into a category orthogonalized feature space;

[0014] Center point calculation unit: calculates the category center point of the feature vector after projection;

[0015] Normalization and row concatenation unit: Normalizes the category center points and concatenates the category center rows into a matrix, which serves as the initial weight vector of the second category classifier New Classifier.

[0016] Furthermore, the first detector includes a backbone feature extractor CNN, a region proposal network RPN, a region proposal pooler RCNN, and a first category classifier Original Classifier; the backbone feature extractor CNN is used to extract the feature map Feature map of the input image, the region proposal network RPN is used to generate candidate boxes to obtain feature sub-map Proposal, the region proposal pooler RCNN pools feature sub-maps Proposal of different sizes into the same size, and the first category classifier Original Classifier determines the category of the feature sub-map Proposal output by the region proposal pooler RCNN.

[0017] Furthermore, fine-tuning the second detector on the new class includes:

[0018] Freeze the parameters of the backbone feature extractor CNN, the region proposal network RPN, and the region proposal pooler RCNN, and fine-tune the parameters of the first category classifier Original Classifier, the feature category orthogonalization module, and the second category classifier NewClassifier; or:

[0019] Freeze the parameters of the first-category classifier Original Classifier, the feature category orthogonalization module, and the second-category classifier New Classifier, and fine-tune the parameters of the backbone feature extractor CNN, the region proposal network RPN, and the region proposal pooler RCNN.

[0020] Furthermore, the first detector also includes a target position refinement module Regressor, which is used to perform regression calculation on the candidate box of the feature sub-image Proposal output by the region proposal pooler RCNN to obtain an accurate shape and position;

[0021] When fine-tuning the second detector, the parameters of the target position refinement module Regressor are fine-tuned or frozen together with the parameters of the first category classifier Original Classifier.

[0022] Furthermore, the category orthogonalization unit specifically includes:

[0023] There are N new categories, each category has K target instances, To represent the original input image of the kth target instance of the nth category, the feature vector output from the first category classifier Original Classifier after the first detector is:

[0024]

[0025] Compute the center point of the eigenvector of the first class:

[0026]

[0027] All eigenvectors of the first class remain unchanged, and the eigenvectors of other classes undergo class orthogonal projection changes without loss of generality. For the jth eigenvector of the i-th class, the eigenvector after projection is

[0028]

[0029]

[0030] ...

[0032]

[0033] Where i = 2, 3, 4, ..., N,<f,a> represents the inner product of vectors f and a;

[0034] The center point calculation unit includes:

[0035] For the nth category, the category center point after projection is:

[0036]

[0037] The normalization and row splicing unit includes:

[0038] For the nth category center point υ n , perform normalization:

[0039]

[0040] The normalized category center points are concatenated to obtain the weight matrix W:

[0041] W=(e 1 ,e 2 ,e 3 ,...e, N ) T

[0042] The second category classifier New Classifier is specifically:

[0043]

[0044] in, Represents the category probability distribution after the second category classifier New Classifier.

[0045] The beneficial effects of the present invention are:

[0046] In an exemplary embodiment of the present invention, based on the characteristics that the model trained on the base class still has class clustering and one-hot encoding orthogonality for samples of the new class (the dataset is divided into the base class with a large amount of labeled data and the new class with a sparse amount of data), the classifier is redesigned and a small sample target detection method based on hot start and feature space class orthogonalization is proposed:

[0047] Using the category orthogonalization method, a new classifier layer (the second category classifier NewClassifier) ​​is added to the last classifier layer of the first detector pre-trained on the base class (the first category classifier Original Classifier). The role of the new classifier layer is a category orthogonalization projection transformation based on Schmidt orthogonalization, which realizes the transition of the second detector when migrating from the base class to the new class, cleverly solving the problems of inconsistent category number and content misalignment in the existing small sample target detection method based on transfer learning.

[0048] At the same time, since the weights of the new classifier layer are obtained by offline calculation through the weights of the pre-trained original classifier, there is no need to randomly initialize random values ​​and then obtain them through a large number of training rounds, which reduces the computational overhead, solves the "cold start" problem of existing detectors from the base class to the new class, and improves the migration efficiency.

[0049] In addition, the random initialization strategy used by existing detectors when migrating to new classes can severely affect the gradients of the classifier's previous modules, such as the backbone, RCNN-head, and RPN, at the beginning of fine-tuning. This can severely damage the numerical distribution of network parameters that have achieved good performance after training on a large amount of labeled data from the base class, severely reducing the detector's generalization and causing it to overfit on specific instances of the new class, significantly reducing its performance. However, using the class orthogonalization method mentioned in this exemplary embodiment to give the new classifier a good initial value can greatly reduce this impact. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A flowchart of a small sample target detection method based on feature space class orthogonalization provided in an exemplary embodiment of the present invention;

[0051] Figure 2 A schematic structural diagram of a first detector provided in an exemplary embodiment of the present invention;

[0052] Figure 3 It is a schematic structural diagram of a second detector provided in an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0053] The technical solution of the present invention is described clearly and completely below with reference to the accompanying drawings. It is apparent that the embodiments described are only a portion of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0054] In the description of the present invention, it should be noted that the directions or positional relationships indicated by terms such as "center," "up," "down," "left," "right," "vertical," "horizontal," "inside," and "outside" are based on the directions or positional relationships described in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limitations on the present invention. Furthermore, terms such as "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0055] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention.

[0056] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0057] See also Figure 1 , Figure 1 A flowchart of a small sample target detection method based on feature space class orthogonalization provided by an exemplary embodiment of the present invention is shown, comprising the following steps:

[0058] The first detector (such as Figure 2 As shown), pre-train the first detector on the base class;

[0059] like Figure 3 As shown, a second detector is constructed by connecting a newly added second class classifier NewClassifier to the end of the first detector, and the second detector is fine-tuned and trained on the new class, where the initial weight vector of the second class classifier NewClassifier is determined by the feature category orthogonalization module;

[0060] Use the trained second detector to perform object detection on the newly input input image;

[0061] The feature category orthogonalization module includes:

[0062] Category orthogonalization unit: performs category orthogonal projection on the feature vector output by the first detector, mapping samples of each category into a category orthogonalized feature space;

[0063] Center point calculation unit: calculates the category center point of the feature vector after projection;

[0064] Normalization and row concatenation unit: Normalizes the category center points and concatenates the category center rows into a matrix, which serves as the initial weight vector of the second category classifier New Classifier.

[0065] Specifically, in this exemplary embodiment, based on the characteristics that the model trained on the base class still has class clustering and one-hot encoding orthogonality for samples of the new class (the dataset is divided into the base class with a large amount of labeled data and the new class with a small amount of data), the classifier is redesigned and a small sample target detection method based on hot start of feature space class orthogonalization is proposed:

[0066] Using the category orthogonalization method, a new classifier layer (the second category classifier NewClassifier) ​​is added to the last classifier layer of the first detector pre-trained on the base class (the first category classifier Original Classifier). The role of the new classifier layer is a category orthogonalization projection transformation based on Schmidt orthogonalization, which realizes the transition of the second detector when migrating from the base class to the new class, and cleverly solves the problems of inconsistent category number and misaligned content in the existing small sample target detection method based on transfer learning (because the number of categories of the new class is inconsistent with that of the base class, and the content does not overlap, the above problems will occur when the model migrates from the base class to the new class; the method of this exemplary embodiment is to add a layer of change to the original classifier, and the dimension of the output probability vector corresponds to the number of new categories. The meaning represented by each dimension of the probability vector also corresponds to a specific category of the new class. Moreover, since the weight initialization of the new classifier is given, the cold start problem is avoided).

[0067] Among them: the pre-trained first detector is used as a special feature extractor, and its output result is used as the feature vector of the sample; and the feature category orthogonalization module is based on Schmidt orthogonalization, and the weight vector of the classifier learned on the base class is orthogonalized and projected. After projection, each dimension of the feature space corresponds to each category on the new class, and the projected weight matrix is ​​used as the initialization weight of the new classifier (the second category classifier New Classifier) ​​for fine-tuning, thereby solving the problem of cold start of the classification head of the small sample detector based on the transfer learning method during fine-tuning.

[0068] At the same time, since the weights of the new classifier layer are obtained by offline calculation through the weights of the pre-trained original classifier, there is no need to randomly initialize random values ​​and then obtain them through a large number of training rounds, which reduces the computational overhead, solves the "cold start" problem of existing detectors from the base class to the new class, and improves the migration efficiency.

[0069] In addition, the random initialization strategy used by existing detectors when migrating to new classes can severely affect the gradients of the classifier's previous modules, such as the backbone, RCNN-head, and RPN, at the beginning of fine-tuning. This can severely damage the numerical distribution of network parameters that have achieved good performance after training on a large amount of labeled data from the base class, severely reducing the detector's generalization and causing it to overfit on specific instances of the new class, significantly reducing its performance. However, using the class orthogonalization method mentioned in this exemplary embodiment to give the new classifier a good initial value can greatly reduce this impact.

[0070] The input of the entire method is a RGB three-channel color image. After processing by the entire system, the category of the target object contained in the image (and precise location information, preferably an exemplary embodiment) will be predicted and generated.

[0071] It should be noted that the selection of base classes has already had Benchmark standards in the field of small sample target detection, which will not be elaborated here.

[0072] More preferably, in an exemplary embodiment, Figure 2 and Figure 3 As shown, the first detector includes a backbone feature extractor CNN, a region proposal network RPN, a region proposal pooler RCNN, and a first category classifier Original Classifier; the backbone feature extractor CNN is used to extract the feature map Feature map of the input image, the region proposal network RPN is used to generate candidate boxes to obtain feature sub-graph Proposal, the region proposal pooler RCNN pools feature sub-graphs Proposal of different sizes into the same size, and the first category classifier Original Classifier determines the category of the feature sub-graph Proposal output by the region proposal pooler RCNN.

[0073] More preferably, in an exemplary embodiment, fine-tuning the second detector on the new class includes:

[0074] Freeze the parameters of the backbone feature extractor CNN, the region proposal network RPN, and the region proposal pooler RCNN, and fine-tune the parameters of the first category classifier Original Classifier, the feature category orthogonalization module, and the second category classifier NewClassifier; or:

[0075] Freeze the parameters of the first-category classifier Original Classifier, the feature category orthogonalization module, and the second-category classifier New Classifier, and fine-tune the parameters of the backbone feature extractor CNN, the region proposal network RPN, and the region proposal pooler RCNN.

[0076] Specifically, in this exemplary embodiment, after the base class training is completed, the pre-trained model weights are saved. When fine-tuning training is performed on the new class, the above two fine-tuning strategies are provided as reference.

[0077] Fine-tuning training can specifically include: using the weight parameters of the pre-trained model on the base class as the initial point of the weight in the fine-tuning stage, using a small sample size of training data on the new class as input, using SGD for gradient backpropagation, and not updating the weights of the frozen layers, but only updating the weight parameters of the fine-tuning layers.

[0078] More preferably, in an exemplary embodiment, the first detector further includes a target position refinement module Regressor, which is configured to perform regression calculation on the candidate box of the feature sub-image Proposal output by the region proposal pooler RCNN to obtain an accurate shape and position;

[0079] When fine-tuning the second detector, the parameters of the target position refinement module Regressor are fine-tuned or frozen together with the parameters of the first category classifier Original Classifier.

[0080] More preferably, in an exemplary embodiment, the class orthogonalization unit specifically includes:

[0081] There are N new categories, each category has K target instances, To represent the original input image of the kth target instance of the nth category, the feature vector output from the first category classifier Original Classifier after the first detector is:

[0082]

[0083] Compute the center point of the eigenvector of the first class:

[0084]

[0085] All eigenvectors of the first class remain unchanged, and the eigenvectors of other classes undergo class orthogonal projection changes without loss of generality. For the jth eigenvector of the i-th class, the eigenvector after projection is

[0086]

[0087]

[0088] ...

[0090]

[0091] Where i = 2, 3, 4, ..., N,<f,a> represents the inner product of vectors f and a;

[0092] The center point calculation unit includes:

[0093] For the nth category, the category center point after projection is:

[0094]

[0095] The normalization and row splicing unit includes:

[0096] For the nth category center point υ n , perform normalization:

[0097]

[0098] The normalized category center points are concatenated to obtain the weight matrix W:

[0099] W=(e 1 ,e 2 ,e 3 ,...e, N ) T

[0100] The second category classifier New Classifier is specifically:

[0101]

[0102] in, Represents the category probability distribution after the second category classifier New Classifier.

[0103] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will readily appreciate that other variations or modifications can be made based on the above descriptions. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications derived therefrom remain within the scope of protection of the present invention.

Claims

1. A small sample target detection method based on feature space class orthogonalization, characterized by: The following steps are involved: Constructing a first detector based on the Faster-RCNN model and pre-training the first detector on the base class; A second detector is constructed by connecting a newly added second-category classifier New Classifier to the end of the first detector, and fine-tuning the second detector on the new class. The initial weight vector of the second-category classifier New Classifier is determined by the feature category orthogonalization module. Use the trained second detector to perform object detection on the newly input input image; The feature category orthogonalization module includes: Category orthogonalization unit: performs category orthogonal projection on the feature vector output by the first detector, mapping samples of each category into a category orthogonalized feature space; Center point calculation unit: calculates the category center point of the feature vector after projection; Normalization and row concatenation unit: Normalizes the category center points and concatenates the category center rows into a matrix, which serves as the initial weight vector of the second category classifier New Classifier.

2. The small sample target detection method based on feature space class orthogonalization according to claim 1, characterized in that: The first detector includes a backbone feature extractor CNN, a region proposal network RPN, a region proposal pooler RCNN, and a first category classifier Original Classifier; the backbone feature extractor CNN is used to extract the feature map Feature map of the input image, the region proposal network RPN is used to generate candidate boxes to obtain feature sub-map Proposal, the region proposal pooler RCNN pools feature sub-maps Proposal of different sizes into the same size, and the first category classifier Original Classifier determines the category of the feature sub-map Proposal output by the region proposal pooler RCNN.

3. The small sample target detection method based on feature space class orthogonalization according to claim 2, characterized in that: The fine-tuning training of the second detector on the new class includes: Freeze the parameters of the backbone feature extractor CNN, the region proposal network RPN, and the region proposal pooler RCNN, and fine-tune the parameters of the first category classifier Original Classifier, the feature category orthogonalization module, and the second category classifier New Classifier; or: Freeze the parameters of the first-category classifier Original Classifier, the feature category orthogonalization module, and the second-category classifier New Classifier, and fine-tune the parameters of the backbone feature extractor CNN, the region proposal network RPN, and the region proposal pooler RCNN.

4. The small sample target detection method based on feature space class orthogonalization according to claim 2 or 3, characterized in that: The first detector also includes a target position refinement module Regressor, which is used to perform regression calculation on the candidate box of the feature sub-image Proposal output by the region proposal pooler RCNN to obtain an accurate shape and position; When fine-tuning the second detector, the parameters of the target position refinement module Regressor are fine-tuned or frozen together with the parameters of the first category classifier Original Classifier.

5. The small sample target detection method based on feature space class orthogonalization according to claim 1, characterized in that: The category orthogonalization unit specifically includes: There are N new categories, each category has K target instances, To represent the original input image of the kth target instance of the nth category, the feature vector output from the first category classifier Original Classifier after the first detector is: Compute the center point of the eigenvector of the first class: All eigenvectors of the first class remain unchanged, and the eigenvectors of other classes undergo class orthogonal projection changes without loss of generality. For the jth eigenvector of the i-th class, the eigenvector after projection is ... Where i = 2, 3, 4, ..., N,<f,a> represents the inner product of vectors f and a; The center point calculation unit includes: For the nth category, the category center point after projection is: The normalization and row splicing unit includes: For the nth category center point υ n , perform normalization: The normalized category center points are concatenated to obtain the weight matrix W: W=(ie 1 ,have been 2 ,have been 3 ,...have been, N ) T The second category classifier New Classifier is specifically: in, Represents the category probability distribution after the second category classifier New Classifier.

Images