A target detection system and method for feature semantic guidance in a data sparse scene

Abstract

The application discloses a feature semantic guidance target detection system and method in a data sparse scene, and belongs to the technical field of pattern recognition and information processing. The system comprises a backbone feature extraction module, a region proposal module for obtaining a foreground proposal region, a region proposal pooling module, and a category target refining module for outputting a target category and position. The backbone feature extraction module comprises a front-end backbone network and a first feature mapper for obtaining image category sensitive features. The region proposal module comprises a second feature mapper for obtaining image extreme sensitive features. The application adopts a shared backbone low-level feature extraction module to obtain low-dimensional semantic features of an image, thereby reducing the model calculation amount. Two feature mappers are added respectively to realize multi-task decoupling of features of different tasks, thereby avoiding representation conflicts among the multi-tasks.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of pattern recognition and information processing, and relates to scene perception and target detection in a data sparse scene in the field of computer vision, and in particular to a target detection system and method guided by feature semantics in a data sparse scene. BACKGROUND

[0002] With the development of computer vision, target detection technology is becoming mature, from manual features to deep learning automatic feature extraction, the accuracy of target detection has been greatly improved. With the gradual application of the Transformer method in various visual fields, the demand for computing power and data for visual target detection is also increasing, which limits the application of detection models in many extremely sparse data scenes or high-cost annotation fields. At the same time, directly using the method of transfer learning to make the model learn in the extremely sparse data scene has a great bottleneck in performance. On the other hand, humans can usually learn some other objects they have never seen before quickly after learning from a small number of samples using some past experience. Studying target detection in a data sparse scene can reduce the dependence of the model on data, improve the generalization ability of the model, and promote the industrial application of the algorithm model.

[0003] Since the RCNN series algorithm was proposed, deep learning models have gradually replaced traditional machine vision methods and become the mainstream algorithm in the field of target detection. The target detection algorithm based on deep learning has developed into two technical routes, Anchor-Based method and Anchor-Free method. Anchor refers to a set of predefined rectangular boxes, representing a priori knowledge of the target distribution aspect ratio and size. The current mainstream target detection method is essentially to classify and regress some candidate regions. In the Anchor-Based method, different anchors generated by sliding windows are candidate regions, for example, FasterRCNN, SSD, YOLOv2 are classic Anchor-Based methods; Anchor-Free abandons the concept of Anchor, and instead calculates classification and regression based on some feature points, among which CornerNet, ExtremeNet, CenterNet, FCOS are representative methods of Anchor-Free. With the gradual reduction of the class imbalance problem in Anchor-Free by some methods such as Focal Loss, Anchor-Free methods gradually can rival Anchor-Based methods in precision.

[0004] To alleviate the dependence on data and promote the performance of the model in the data sparse scene, few-shot object detection aims to learn from a few labeled instances in the unseen class domain during the training phase. According to the difference in the training method of the few-shot object detection method, the existing few-shot object detection methods are divided into meta-learning methods and transfer learning methods. Meta-learning method, also known as "learning to learn", is proposed to solve the problem of poor generalization performance and poor adaptability to new class tasks of traditional neural network model. Few-shot object detection based on meta-learning mainly encodes the support image and the query image using a twin architecture, and then performs dense visual feature matching between the query image features and the support image features to perform visual object detection. Few-shot object detection based on transfer learning is usually built on Faster RCNN, and adopts the pre-training and fine-tuning paradigm to solve the problem of data sparsity. Compared with meta-learning methods, few-shot object detection based on transfer learning pays more attention to the knowledge transfer process and is simpler in algorithm implementation. Few-shot object detection has made remarkable progress in the past two years, but the existing methods are far from meeting the actual application, especially in the scene where the sample is extremely sparse, the reasons are as follows:

[0005] The object detection task needs to predict the class of the target and the specific position in the image at the same time, and the specific position is given in the form of pixel coordinates of the object relative to the top left corner of the image. In the two-stage object detection method, the detection network contains two sub-networks: the backbone network and the region proposal network. The preprocessed image is first passed through the feature extraction module of the backbone network to obtain shared image features, and then the shared features are sent to the region proposal network and the subsequent module of the backbone network. For the shared features sent to the region proposal network, the image receptive field is increased through a convolution module, and then the enhanced image features are obtained by two convolution branches to obtain the foreground probability and coarse-grained features. For the shared features sent to the subsequent module of the backbone network, the object features are first cut out from the shared features according to the output of the region proposal network, and then the object features are obtained by region of interest pooling. Finally, the features are used to obtain the fine-grained class and accurate position of the object by using independent classification and regression networks.

[0006] In the above process, the backbone network and the region proposal network are two independent modules, and each module contains two different tasks of classification and regression, which increases the difficulty of network learning. From the implementation level, the image features extracted by the feature extractor in the backbone network need to be provided to the region proposal network and the subsequent modules of the backbone network. The main function of the region proposal network is to extract the foreground object and its rough position, while the backbone network needs to identify the fine-grained category of the object and the accurate position of the image. On the one hand, the foreground and background in this process are relative. In traditional object detection, the foreground category refers to the labeled category in the training data set, and the categories not labeled in the training data set are determined as background, and the categories of the test image and the training image remain the same. In the small sample target detection scene setting, due to the existence of unseen categories, the training data set and the test data set no longer share the category space, which requires the model to adjust the judgment strategy for foreground and background. However, the data available for strategy adjustment is extremely sparse, and part of the sampled data may also have noise, further increasing the learning difficulty, resulting in a significant decrease in the prediction performance of the model in the sparse scene. On the other hand, intuitively, the image features used for foreground / background judgment and fine-grained category judgment must have some differences. For different categories, the feature regions that need to be paid attention to are different. The region proposal network needs to frame the bounding box of the object according to the extreme points of the object, so it pays more attention to the rough position of the extreme points of the object. The backbone network needs to classify the specific category of the object, and needs to pay special attention to some discriminative details of the object. The existing technology directly uses two backbone networks to learn parameters for the region proposal network and the backbone network to balance the differences in image features required for foreground / background judgment and fine-grained category judgment. However, in the image convolution process of feature processing, the low-dimensional semantic features of the image such as texture and color are shared among many categories. If two independent feature extraction processes are used, the model calculation will be increased and the real-time performance will be reduced.

[0007] In summary, how to decouple different tasks to solve the representation conflict between tasks and improve the prediction performance and learning efficiency of the model is a problem that needs to be solved. SUMMARY

[0008] The purpose of the present application is to overcome the problems of the prior art and provide a feature semantic guided target detection system and method in a data sparse scene.

[0009] The object of the present application is achieved by the technical solutions below: a target detection system with feature semantic guidance in a data sparse scene, which specifically comprises a backbone feature extraction module, a region proposal module for obtaining a foreground proposal region, a region proposal pooling module, and a category target refining module for outputting a target category and position, the backbone feature extraction module comprises a front-end backbone network, a first feature mapper for obtaining image category sensitive features, which are sequentially connected, the region proposal module comprises a second feature mapper for obtaining image extreme sensitive features; the front-end backbone network is used for extracting image shallow features, and is connected with the region proposal module; the region proposal pooling module is used for outputting a candidate feature map, and is connected with the first feature mapper, the region proposal module, and the category target refining module.

[0010] In an example, the first feature mapper and the second feature mapper have the same structure, and parameters are not shared.

[0011] In an example, the feature mapper comprises a plurality of sub-feature mappers, each of which comprises a plurality of convolution layers which are sequentially connected, and an input end of a first convolution layer is connected with an output end of a last convolution layer.

[0012] In an example, the region proposal module sequentially comprises the second feature mapper, a large kernel convolution submodule for increasing a feature receptive field, the large kernel convolution submodule is connected with a coarse-grained bounding box regression submodule and a foreground-background determination submodule which are arranged side by side; the coarse-grained bounding box regression submodule is used for predicting a preliminary position (approximate position) of a proposal bounding box, and the foreground-background determination submodule is used for realizing confidence judgment of a foreground target, thereby obtaining a foreground proposal region.

[0013] In an example, the category target refining module comprises a regressor, which outputs only a four-dimensional vector representing horizontal and vertical coordinates of a center point of an image and width and height of a region proposal bounding box for features of a given proposal bounding box.

[0014] In an example, the category target refining module comprises a classifier, which is used for outputting a fine-grained category of a given proposal bounding box.

[0015] It should be further explained that the technical features corresponding to each of the above examples can be combined or replaced to form new technical solutions.

[0016] The present application also comprises a target detection method with feature semantic guidance in a data sparse scene, which is implemented based on the target detection system with feature semantic guidance formed according to any one of the above examples or a combination of multiple examples, and comprises the following steps:

[0017] The front-end backbone network extracts image shallow features;

[0018] The first feature mapper obtains image category sensitive features based on the image shallow features;

[0019] The second feature mapper in the region proposal module acquires extreme value sensitive features based on image shallow layer features, and then outputs foreground proposal regions;

[0020] The region proposal pooling module pools feature maps of different sizes into specific size candidate frame feature maps according to the foreground proposal regions;

[0021] The category target refining module outputs target categories and positions according to image category sensitive features and candidate frame feature maps.

[0022] In an example, the outputting of the foreground proposal regions comprises the following steps:

[0023] Acquiring extreme value sensitive features based on image shallow layer features;

[0024] Increasing the receptive field of the extreme value sensitive features;

[0025] Respectively performing convolution processing on the extreme value sensitive features after increasing the receptive field, and then realizing target candidate frame and target confidence judgment, and then obtaining the foreground proposal regions.

[0026] In an example, after the foreground proposal regions are obtained, the method further comprises:

[0027] Filtering out candidate frames with confidence lower than a threshold, and filtering out unreasonable candidate frames by using a non-maximum suppression algorithm to obtain final foreground proposal regions.

[0028] In an example, the method further comprises a novel category learning step:

[0029] Initializing a novel category classifier by using label semantic similarity between existing categories and the novel category;

[0030] Extracting the most similar base class prototypes corresponding to the new category from a base class prototype library according to a label category similarity matrix to form an initial weight of the new category classifier, only updating the last layer parameters of the classification network in the category target refining module, and fixing the network weight parameters in the detection system;

[0031] Inputting the novel category image into the detection system, and starting parameter updating of the detection system until convergence.

[0032] It should be further explained that the technical features corresponding to the above examples can be combined or replaced to form new technical solutions.

[0033] Compared with the prior art, the present application has the following beneficial effects:

[0034] 1. In an example, partial low-level image semantic feature sharing is achieved by using a backbone feature extraction module, which reduces model calculation and improves model calculation real-time performance; in the high-dimensional image semantic extraction part, two feature mappers are added to decouple the features of different tasks, avoid representation conflict between multiple tasks, and provide image features that meet the current task requirements for foreground / background judgment and fine-grained category judgment, thereby ensuring the detection performance of the system.

[0035] 2. In an example, the region proposal module uses a large core convolution design to explicitly increase the receptive field from the algorithm design level and strengthen the region proposal network's ability to identify objects of different sizes.

[0036] 3. In an example, a lightweight class-agnostic regressor is designed, which will learn the inductive bias of "capturing the extreme region of the object" regardless of the class, improving the regression accuracy; at the same time, from the training point of view, due to the class-agnostic feature, the regressor is free from the problem of needing to retrain a new subnetwork due to the difference in the number of image classes between unseen classes and seen classes, improving the calculation efficiency.

[0037] 4. In an example, in the process of generalizing to data sparse scenarios, the experience of learned classes (text semantic information) is used to assist the learning of unseen classes, which is more in line with human learning intuition, and only a small number of parameters in the last layer need to be fine-tuned during the generalization process, reducing the risk of overfitting of the model on a small number of samples, improving the accuracy of the model, accelerating the convergence efficiency of the algorithm, and achieving efficient learning on unseen classes. BRIEF DESCRIPTION OF DRAWINGS

[0038] The specific embodiments of the application will be described in further detail below with reference to the accompanying drawings, which are used to provide further understanding of the application and form a part of the application. In these drawings, the same reference numerals are used to represent the same or similar parts. The illustrative embodiments of the application and their descriptions are used to explain the application and should not be construed as an improper limitation of the application.

[0039] Figure 1 A detection network block diagram in an example of the application;

[0040] Figure 2 A region proposal pooling module structure diagram in an example of the application;

[0041] Figure 3 A sub-feature mapper structure diagram in an example of the application;

[0042] Figure 4 A region proposal module structure diagram in an example of the application;

[0043] Figure 5 A class weight initialization diagram for a class determination network in an example of the present application. DETAILED DESCRIPTION

[0044] The technical solutions of the present application will be described clearly and completely below with reference to the drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0045] In the description of the present application, it should be noted that the directions or positional relationships indicated by "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like are described based on the directions or positional relationships described in the drawings, and are only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the devices or elements indicated must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the ordinal numbers (such as "first and second", "first to fourth", etc.) are used to distinguish objects, and are not limited to the order, and cannot be understood as indicating or implying relative importance.

[0046] In the description of the present application, it should be noted that, unless otherwise explicitly specified and limited, "mounting", "connection" and "connection" should be understood in a broad sense, for example, it can be fixed connection, or detachable connection, or integral connection; it can be mechanical connection, or electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or the internal communication of two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0047] In addition, the technical features involved in the different embodiments of the present application described below can be combined with each other as long as there is no conflict between them.

[0048] In an example, as shown in Figure 1 A target detection system guided by feature semantics in a data sparse scene, which specifically comprises a backbone feature extraction module, a region proposal module DearRPN, a region proposal pooling module and a class target refining module. The backbone feature extraction module Backbone comprises a front-end backbone network and a first feature mapper connected in sequence; the output end of the front-end backbone network is connected with the region proposal module, the region proposal module is connected with the region proposal pooling module, and the region proposal pooling module is connected with the class target refining module.

[0049] Specifically, the front-end backbone network is used to extract shallow features of the input image. For target detection in a sparse scene (small sample target detection), the dataset is generally divided into a base dataset with a large number of samples and a new class dataset with only sparse samples. The sparse scene is simulated by the new class dataset. The front-end backbone network is pre-trained by the images in the base dataset. After training based on the large sample base dataset, learning is performed on the new class dataset with sparse data to achieve target detection in a sparse scene, i.e., the input image is an image sample in the base dataset and the new class dataset. As an option, the first three blocks of ResNet-101 pre-trained on ImageNet-1k are used as the backbone network, and the image of the learning class is input into the backbone network to obtain a shallow feature map. The backbone network mapper uses a Res Block with the same structure as the ResNet-101 Block 4 as a feature mapper, which is composed of 23 sub-structures of the same Bottleneck structure. For an input image with a size of HxWx3, the image semantic features of the shallow feature extractor are The process will downsample the image, and the feature Of course, the front-end backbone network can also be other feature extraction networks, such as VGG network, etc. In this example, a backbone feature extraction module is used to share part of the low-level image semantic features, which reduces the model calculation amount and improves the calculation real-time performance of the model.

[0050] Specifically, the first feature mapper is used to obtain image class-sensitive features, i.e., feature information that meets fine classification, which facilitates the subsequent class target refinement module to accurately identify the foreground target based on the class-sensitive features. The structure of the first feature mapper can be the same as a single sub-structure of the front-end backbone structure.

[0051] Specifically, as shown in Figure 2 The region proposal pooling module, i.e., the backbone network head RCNN Head, is used to cut out the corresponding feature region from the feature map output by the backbone network feature mapper according to the position output by the region proposal pooling module, and pool it into an equal-sized candidate feature map. The region proposal pooling module is connected to the output end of the region proposal module and the first feature mapper, and is connected to the input end of the class target refinement module.

[0052] Specifically, the region proposal module includes a second feature mapper for obtaining image extreme value sensitive features, which can have the same structure as a single substructure of the front-end backbone structure. The second feature mapper is used to extract extreme value sensitive features, i.e., feature information conducive to candidate box region division, to facilitate the subsequent class target refining module to identify the region where the target is located based on the extreme value sensitive features, i.e., output the best proposal region. In the high-dimensional image semantic extraction part in this example, two feature mappers are added respectively to implement multi-task decoupling of different features, avoid representation conflict between multi-tasks, and provide image features meeting the current task requirements for foreground / background judgment and fine-grained class determination, so as to ensure the detection performance of the system.

[0053] Specifically, the class target refining module includes a class discrimination network and a position refining network to output class information L RP-cls of the target and accurate position information L RP-reg of the target. It should be noted that the specific structures of the backbone feature extraction module, the region proposal module, the region proposal pooling module, and the class target refining module can be known by those skilled in the art based on common knowledge, and the network structure disclosed in a small sample target detection method based on a double-branch region proposal network with publication number CN114743045A and the name can be referred to for specific reference.

[0054] The present application extracts primary features of an image through a shallow layer of a backbone network, then maps the feature map to two feature spaces of edge sensitivity and class sensitivity respectively by using two independent feature space mapping modules with the same structure, then sends the edge sensitive features to a region proposal network (region proposal module) to extract the coarse-grained position of the object of interest, and obtains the class sensitive features of the corresponding object from the class sensitive feature space according to the coarse-grained position to perform class judgment and regression box refining.

[0055] The present application

[0056] In an example, the first feature mapper and the second feature mapper have the same structure and do not share parameters, including all learnable parameters, and the two feature mappers are updated in gradient respectively.

[0057] In an example, the feature mapper includes a plurality of sub-feature mappers, each sub-feature mapper includes a plurality of convolutional layers connected in sequence, and the input end of the first convolutional layer is connected to the output end of the last convolutional layer. Preferably, taking the structure of the second feature mapper as an example, the extreme sensitive feature mapper (second feature mapper) input into the region proposal network is stacked 23 times with the same substructure as the backbone network, and the specific structure of the sub-feature mapper is as follows Figure 3As shown, it includes three sequentially connected convolutional layers, and the input end of the first convolutional layer is connected with the output end of the last convolutional layer, but the structure parameters are separated from the backbone network and are respectively updated. Therefore, since the module is independent of the first feature mapper in the backbone network, the features after the module are decoupled from the features of the backbone network in high-dimensional semantics, i.e., the class-sensitive features and the extreme value-sensitive features of the input image are respectively obtained through the first feature mapper and the second feature mapper.

[0058] In an example, as shown in FIG. 2, the region proposal module sequentially includes the second feature mapper, the large kernel convolutional sub-module, the coarse-grained bounding box regression sub-module and the foreground-background judgment sub-module. Figure 4 As shown, the region proposal module sequentially includes the second feature mapper, the large kernel convolutional sub-module, the coarse-grained bounding box regression sub-module and the foreground-background judgment sub-module. Specifically, the second feature mapper is a feature mapping module consistent with the fourth Block structure of ResNet-101, to obtain the extreme value-sensitive features. The large kernel convolutional sub-module is a group of large kernel convolutional layers with a kernel size of 15x15, to increase the feature receptive field. Specifically, in this example, the 15x15 convolution is split into a group of 15x1 and 1x15 two convolutions, which greatly reduces the parameter amount without losing the model accuracy. The large kernel convolution design explicitly increases the receptive field from the algorithm design level, and strengthens the recognition ability of the region proposal network for objects of different sizes. The coarse-grained bounding box regression sub-module and the foreground-background judgment sub-module have the same structure, which are 1x1 convolutional layers. The coarse-grained bounding box regression sub-module is used to predict the candidate box of the target, and the foreground-background judgment sub-module is used to realize the confidence judgment of the foreground target, to combine the foreground proposal region with the predefined anchor box parameters to calculate the actual object of interest proposal region. In summary, the data processing process of the region proposal module is as follows:

[0059] The low-level semantic features are input into the region proposal network, the extreme value-sensitive features are obtained through feature mapping, and then a group of large kernel convolutions (large kernel convolutional sub-module) in the region proposal network are used to map the extreme value-sensitive features to increase the receptive field. The mapped features are respectively input into two 1x1 convolutions in the feature space (one of which has an output channel of 1, representing the foreground probability; the other has an output channel of 4, representing the coordinate offset) to obtain the coarse-grained proposal region of the object of interest. The loss function in this process is as follows:

[0060]

[0061]

[0062] L RPN =λL RP_reg +L RP_cls

[0063] Wherein, p * represents the label of the i-th region proposal box; o* The true label represents the offset value of the i-th region suggestion box, i.e., the offset of the true box relative to the anchor box; N cls N represents the number of samples for the input region proposal classifier. reg This represents the total number of anchor boxes; λ is a hyperparameter that controls the contribution of regression and classification to the final weights, because N reg Usually with N cls Since the ratio is 10, this invention follows the traditional Faster R-CNN by setting λ to 10 to balance the loss magnitudes of regression and classification.

[0064] In one example, the region proposal pooling module includes a pooling submodule that, when cropping the original features of the image from the base feature map based on the obtained foreground proposal box positions, pools the base features of candidate boxes of different sizes into proposal box feature maps of the same size. Then, the proposal box feature maps are flattened and fed into the category discrimination network and the location refinement network to obtain the precise location and category of the object.

[0065] In one example, the category target refinement module includes a lightweight class-independent regressor that, for a given candidate box with features f rp The regressor outputs only a four-dimensional vector representing the x and y coordinates of the image center point and the width and height of the region candidate box. The original Faster R-CNN, during the region proposal module's regression, regresses four offset values ​​{o1, o2, ..., o...} for each category of each region proposal candidate box. i}={(x1,y1,w1,h1),(x2,y2,w2,h2),...,(x i ,y i ,w i ,h i )} represent the coordinates of the center point of the image and the width and height of the image, respectively. Since the inductive bias of the regression branch for the regression offset value is the extreme value region of the object in the query region suggestion candidate box in the image, a class-independent regressor can be designed to calculate and correct the fine regression offset value of each region suggestion candidate box. Specifically, this invention replaces the original bulky regressor with a lightweight class-independent regressor, which, for a given region suggestion candidate box feature f rp Regardless of the number of classes in the training or testing samples, this module outputs only a single four-dimensional vector o. * =(x * ,y * ,w * ,h *) represent the horizontal and vertical coordinates of the center point of the image, the width and height of the region proposal box. Therefore, regardless of any category, the regressor of the target detection region proposal module learns the inductive bias of "capturing the extreme region of the object". At the same time, from the perspective of training, due to the category-independent feature, the region proposal module is free from the problem of needing to retrain a new subnetwork due to the difference in the number of image categories between unseen categories and seen categories, improving the computational efficiency.

[0066] In an example, the category target refining module includes a classifier for outputting the fine-grained category of a given candidate box.

[0067] Combining the above examples, a preferred detection framework, i.e., a detection system, of the present application is obtained, which improves the traditional FasterRCNN. The region proposal network and the region features of the backbone branch are projected into different feature spaces through two independent feature mapping modules with the same structure. The features input into the region proposal network are extracted using a large kernel convolution layer of 15x1 and 1x15 to obtain edge-sensitive features with a larger receptive field. Then, the edge-sensitive features are sent into the large kernel region proposal network to extract the coarse-grained position of the object of interest.

[0068] The present application also includes a feature semantic guided target detection method in a data sparse scene, which is implemented based on the target detection system formed by any one of the above examples or a combination of multiple examples. The pre-training stage of the detection method includes the following steps:

[0069] S1': the front-end backbone network extracts image shallow features;

[0070] S2': the first feature mapper obtains image category-sensitive features based on the image shallow features;

[0071] S3: the second feature mapper in the region proposal module obtains extreme-sensitive features based on the image shallow features, and then outputs foreground proposal regions;

[0072] S4': the region proposal pooling module outputs candidate box feature maps according to the foreground proposal regions;

[0073] S5': the category target refining module outputs the target category and position according to the image category-sensitive features and the candidate box feature maps.

[0074] Preferably, steps S2 and S3 are executed in parallel to improve the computational efficiency of the algorithm. After completing the pre-training of the detection system based on the above method, the target detection is realized based on the same pre-training method as above.

[0075] Preferably, step S1' further includes:

[0076] The dataset is divided into class-disjoint learning classes and unseen classes, and some instances are randomly sampled from the unseen classes as test images. Then N classes are extracted from the unseen classes, and K images are extracted from each class to simulate a sparse data scenario.

[0077] In an example, outputting the foreground proposal region comprises the following steps:

[0078] S31': obtaining extreme value sensitive features based on image shallow features;

[0079] S32': increasing the receptive field of the extreme value sensitive features;

[0080] S33': respectively performing convolution processing on the extreme value sensitive features after increasing the receptive field, thereby realizing target candidate frame and target confidence judgment, and thereby obtaining the foreground proposal region.

[0081] In an example, after obtaining the foreground proposal region, the following steps are further included:

[0082] For each feature map, the candidate frame with a confidence lower than a threshold is filtered out, and a non-maximum suppression algorithm is used to filter out unreasonable proposal frames with a boundary exceeding the edge of the image and small object frames with an area smaller than a preset threshold. Finally, the proposal frame higher than the threshold in the remaining proposal frame is returned and output to the pooling submodule.

[0083] The target detection system based on feature semantic guidance in a sparse data scenario, when applied to a generalization learning process (knowledge transfer process) to simulate an unseen class learning process in a sparse data environment, uses a pre-trained ResNet-101 on ImageNet-1k as a backbone feature extractor. In the pre-training stage, the parameters of the feature extractor except the last two layers are frozen, the other parts of the model are randomly initialized, and forward propagation and back propagation are performed. The unseen class learning includes the following steps:

[0084] S1": label semantic guided initialization: as shown in Figure 5 , first calculate the similarity matrix SimilarityM sim between the unseen classes and the learning classes to obtain the most similar learning class index W novel of each unseen class; the last layer of the class determination network of the detection network (detection system) in the pre-training process is taken out, and the weights corresponding to the most similar learning class index of the class are taken out in the order of the unseen classes to form a new weight matrix as the weights of the last layer of the class determination network, and the bias is set to 0. Specifically, the last layer weight matrix W baseThe base class prototype bank is regarded as a base class prototype bank, and then the corresponding prototype is extracted from the prototype bank as the initial prototype of the new class according to the base class semantic similarity. For similarity measurement, the pre-trained language pair class label is used for word embedding, and in the application, the pre-trained GloVe is used to convert the class labels of the new class and the base class into word vectors respectively, and the similarity is measured by cosine, to obtain the label semantic similarity matrix M of the base class and the new class sim :

[0085] M sim =sim(glv(C base ),glv(C novel ))

[0086] Wherein, glv(·) represents a word embedding operation.

[0087] S2'': according to the label category similarity matrix M sim The most similar base class prototype corresponding to the new class is extracted from the base class prototype bank to form the initial weight of the new class classifier, and the weight parameters of the entire detection network except the last layer parameters of the classification network are fixed and no longer updated.

[0088] S3'': the unseen class image is input into the detection network, so that the detection network starts parameter updating until convergence.

[0089] In the process of generalizing to the unseen class in the training stage, the model first calculates the semantic similarity of the seen class and the unseen class using the pre-trained class label semantic relationship on WordNet, and according to the semantic similarity, the initial value of the unseen class is obtained by multiplying the similarity value of the most similar seen class prototype in the last layer of the original algorithm classification layer classification network, and then the initial value is used to fine-tune only the last layer classification network on a small amount of data labeling, so as to obtain good generalization performance of the unseen class.

[0090] To verify the superiority of the feature decoupling method of the application, the method (pre-training + knowledge transfer + target detection) described in the application is experimented on Pascal VOC, wherein DR represents the RPN of feature decoupling (including fine-tuning strategy), CAR represents the class-independent regressor, and RBC represents the label semantic guided initialization, and the experimental results are shown in the following table 1:

[0091] Table 1 Experimental results on Pascal VOC dataset

[0092]

[0093] As can be seen from the table, after adding the feature decoupling, the detection capability for the base class image is improved to a certain extent (AP50 rises by 1 point), and the detection capability for the new class image rises by 15-30 points in different experiments, which proves the superiority of the feature decoupling method proposed in the application. In addition, as can be seen from Table 1, after adding the label semantic guidance, the performance of the model is improved by 4-6 points.

[0094] The embodiment provides a storage medium, which, in combination with any one or multiple examples described above, forms a target detection system with feature semantic guidance in a data sparse scene, and has the same inventive concept. The storage medium has computer instructions stored thereon. The computer instructions perform the steps of the target detection system with feature semantic guidance in a data sparse scene formed by any one or multiple examples described above when executed.

[0095] Based on such understanding, the technical solution of the embodiment or the part of the technical solution that essentially contributes to the prior art or the part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes instructions for causing a computer device (which can be a personal computer, a server, or a network device) to execute all or part of the steps of the method described in the embodiments of the application. The storage medium described above includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0096] The application also includes a terminal, which has the same inventive concept as any one or multiple examples corresponding to the target detection system with feature semantic guidance in a data sparse scene. The terminal includes a memory and a processor. The memory has computer instructions stored thereon, which can be executed on the processor. The processor executes the steps of the target detection system with feature semantic guidance in a data sparse scene when executing the computer instructions. The processor can be a single-core or multi-core central processing unit or a specific integrated circuit, or one or more integrated circuits configured to implement the application.

[0097] In an example, the terminal, i.e., the electronic device, is in the form of a general-purpose computing device. The components of the electronic device can include but are not limited to the at least one processing unit (processor), the at least one storage unit, and a bus connecting different system components, including the storage unit and the processing unit.

[0098] The storage unit can include a readable medium in the form of volatile storage such as random access memory (RAM) 3201 and / or cache memory, and also can include non-volatile storage such as read only memory (ROM).

[0099] The storage unit can include a readable medium in the form of volatile storage such as random access memory (RAM) 3201 and / or cache memory, and also can include non-volatile storage such as read only memory (ROM).

[0100] The storage unit also can include a program / utility, having a set (at least one) of program modules that are configured to carry out the functions of the examples described herein, including an operating system, one or more application programs, other program modules, and program data, each or some combination thereof, which may

[0101] The bus can represent one or more of several types of bus structures, including an external bus, memory bus, or memory controller, a peripheral bus, a graphics acceleration port, a local bus, and so forth using any of a variety of bus architectures.

[0102] The electronic device also can communicate with one or more external devices such as a keyboard or a pointing device, through an I / O interface. Additionally, the electronic device can communicate with one or more devices that enable a user to interact with the electronic device, and / or one or more devices (e.g., a router, a modem, a network card, etc.) that enable the electronic device to communicate with one or more other computing devices. Such communication can occur via an I / O interface. Still yet, the electronic device can communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or the Internet) through a network adapter. It should be appreciated that the electronic device can be a part of a larger system, including but not limited to a distributed computing environment, a grid computing environment, a cloud computing environment, and / or any other working environment in which a task can be performed by a remote processing device. The electronic device can also be a client device in a client-server environment.

[0103] Through the above description, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the example embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash disk, a mobile hard disk, or the like) or a network, and includes a number of instructions to make a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) execute the methods of the example embodiments.

[0104] The above detailed description is a detailed description of the present application, and cannot be considered as limiting the specific embodiments of the present application to these descriptions. For those skilled in the art, without departing from the concept of the present application, a number of simple deductions and substitutions can be made, which should be considered as falling within the scope of protection of the present application.

Claims

1. A target detection system with feature semantic guidance in a data sparse scenario, comprising a backbone feature extraction module, a region proposal module for obtaining foreground proposal regions, a region proposal pooling module, and a class target refining module for outputting target classes and positions, characterized in that: The main feature extraction module comprises a front-end main network, a first feature mapper for obtaining image class sensitive features, and a region proposal module comprising a second feature mapper for obtaining image extreme value sensitive features; The front-end main network is used for extracting image shallow features and is connected with the region proposal module; the region proposal pooling module is used for outputting candidate feature maps and is connected with the first feature mapper, the region proposal module, and the class target refining module; The first feature mapper and the second feature mapper have the same structure and do not share parameters; the feature maps are respectively mapped into edge sensitive and class sensitive feature spaces through the first feature mapper and the second feature mapper; then the edge sensitive features are sent into the region proposal module to extract the coarse-grained position of the object of interest; the class sensitive features corresponding to the object are obtained from the class sensitive feature space according to the coarse-grained position, and class judgment and regression box refining are performed; The coarse-grained position of the object of interest comprises: A group of large kernel convolutions in the region proposal module perform feature mapping on the image extreme value sensitive features to increase the receptive field; the mapped features are respectively passed through two 1x1 convolutions in the feature space to obtain coarse-grained proposal regions of the object of interest; one of the two 1x1 convolutions has one output channel, which represents the foreground probability; the other has four output channels, which represent the coordinate offset. The class target refining module comprises a regressor, which is independent of the class and outputs a four-dimensional vector representing the horizontal and vertical coordinates of the center point of the image and the width and height of the region candidate box for the given candidate box feature. 2.The system of claim 1, wherein: The feature mapper comprises a plurality of sub-feature mappers, each of which comprises a plurality of convolution layers connected in sequence, and the input end of the first convolution layer is connected with the output end of the last convolution layer. The region proposal module comprises the second feature mapper and a large kernel convolution submodule in sequence, and the large kernel convolution submodule is connected with a coarse-grained bounding box regression submodule and a foreground-background judgment submodule arranged side by side; 3.The system of claim 1, wherein: The coarse-grained bounding box regression submodule is used for predicting the preliminary position of the proposal candidate box, and the foreground-background judgment submodule is used for realizing the confidence judgment of the foreground target to obtain the foreground proposal region. The class target refining module comprises a classifier for outputting the fine-grained class of the given candidate box. 4.The system of claim 1, wherein: The method comprises the following steps:

5. A method for target detection under data sparse scenarios with feature semantic guidance, based on the target detection system under data sparse scenarios with feature semantic guidance according to any one of claims 1-4, characterized in that: The front-end main network extracts image shallow features; The first feature mapper obtains image class sensitive features based on the image shallow features; The second feature mapper in the region proposal module obtains extreme value sensitive features based on the image shallow features, and then outputs the foreground proposal region; The region proposal pooling module outputs the candidate box feature map according to the foreground proposal region; The class target refining module outputs the target class and position according to the image class sensitive features and the candidate box feature map. The output of the foreground proposal region comprises the following steps: 6.The method of claim 5, wherein the method further comprises: Obtain the extreme value sensitive features based on the image shallow features; Increase the receptive field of the extreme value sensitive features; Convolve the extreme value sensitive features after increasing the receptive field to realize target candidate box and target confidence judgment, and then obtain the foreground proposal region. After obtaining the foreground proposal region, the following steps are further included:

7. The method of claim 6, wherein the method further comprises: ​ The candidate boxes with confidence lower than the threshold are filtered out, and a non-maximum suppression algorithm is used to filter unreasonable candidate boxes to obtain final foreground proposal regions. 8.The method of claim 5, wherein the method further comprises: The method comprises a novel category learning step: The label semantic similarity between the existing categories and the novel category is used to initialize the novel category classifier. According to the label category similarity matrix, the most similar base class prototypes corresponding to the new category are extracted from the base class prototype library to form the initial weight of the new category classifier, so that the last layer parameters of the classification network in the category target refining module are updated, and the network weight parameters in the detection system are fixed; The novel category image is input into the detection system, so that the detection system starts parameter updating until convergence.

Images