Small sample target detection method and device, electronic equipment and storage medium

Abstract

The present application relates to the technical field of artificial intelligence, and provides a small sample target detection method and device, electronic equipment and storage medium, when training a small sample target detection model, the semantic similarity of image level, instance level and pixel level is calculated, and the fusion problem of the semantic similarity of different levels is considered, when the semantic similarity of different levels is fused, the present application does not simply weight and sum different semantic similarities, but fuses through a series of convolution operations, so that the model can obtain more rich semantic information, the robustness of the small sample target detection model obtained by training is stronger, so that the unknown class can be accurately detected by using the small sample target detection model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a small sample target detection method and device, electronic equipment and storage medium. BACKGROUND

[0002] Target detection technology, as one of the most important research directions in the field of computer vision, is widely used in the fields of industry, agriculture, medical treatment and the like. Traditional target detection methods rely on labeled data sets for training, and cannot detect samples of classes not appearing in the labeled data. In actual applications, a large number of classes of samples are unknown classes for the model, and labeling these data one by one will result in huge cost, and therefore how to enable the model to successfully detect the classes of objects without training on unknown data has become a technical problem. SUMMARY

[0003] In view of the above, it is necessary to provide a small sample target detection method, device, electronic equipment and storage medium, which can detect unknown classes without class training.

[0004] A first aspect of the present application provides a small sample target detection method, which comprises:

[0005] Obtaining a plurality of query sample images and a target sample image corresponding to each query sample image, and obtaining target instances belonging to the same class as the query sample image in each target sample image by constructing a small sample target detection model;

[0006] Calculating an image-level semantic similarity based on the candidate region where the target instance is located and the corresponding query sample image;

[0007] Calculating an instance-level semantic similarity based on the candidate region where the target instance is located and the corresponding query sample image;

[0008] Calculating a pixel-level semantic similarity based on the candidate region where the target instance is located and the corresponding query sample image;

[0009] Performing similarity fusion based on the candidate features of the candidate region, the image-level semantic similarity, the instance-level semantic similarity and the pixel-level semantic similarity to obtain a semantic similarity fusion result;

[0010] Training the small sample target detection model based on the semantic similarity fusion result;

[0011] Using the trained small sample target detection model to detect regions in a to-be-detected target image that belong to the same class as the instance of a to-be-detected query image.

[0012] In an optional implementation, the training of the small sample target detection model based on the semantic similarity fusion result comprises:

[0013] The small sample target detection model outputs a predicted class label of the target instance in each of the target sample images and a predicted bounding box of the target instance, and the predicted bounding box corresponds to the candidate region;

[0014] The plurality of target sample images are divided into positive sample images and negative sample images based on the predicted bounding box;

[0015] For the positive sample images, a preset first loss function is used to calculate a first cross-entropy loss value based on the predicted class label and a true class label of the instance;

[0016] For the negative sample images, a preset second loss function is used to calculate a second cross-entropy loss value based on the predicted class label and the true class label of the instance;

[0017] The total loss value is calculated based on the first cross-entropy loss value and the second cross-entropy loss value;

[0018] A gradient descent algorithm is used to train the small sample target detection model based on the loss value to obtain a trained small sample target detection model.

[0019] In an optional implementation, the target instance in each of the target sample images that belongs to the same class as the target instance in the corresponding query sample image comprises:

[0020] A small sample target detection model based on a R-CNN framework is constructed, and the small sample target detection model comprises a feature extraction network, a feature optimization network, and a candidate region generation network;

[0021] The first sample feature of the target sample image is extracted by the feature extraction network, and the second sample feature of the query sample image is extracted;

[0022] The first sample feature is optimized to obtain a first optimized feature by the feature optimization network, and the second sample feature is optimized to obtain a second optimized feature;

[0023] The target instance in each of the target sample images that belongs to the same class as the target instance in the corresponding query sample image is obtained based on the first optimized feature and the second optimized feature by the candidate region generation network.

[0024] In an optional implementation, the image-level semantic similarity is calculated based on the candidate region where the target instance is located and the corresponding query sample image, and the calculation includes:

[0025] The region feature of the candidate region is obtained;

[0026] The second optimization feature of the query sample image is subjected to an average pooling operation to obtain a first pooling feature;

[0027] The first pooling feature is subtracted from the feature with the same channel identifier in the region feature to obtain a first difference feature;

[0028] The first difference feature is subjected to a convolution operation to obtain the image-level semantic similarity.

[0029] In an optional implementation, the instance-level semantic similarity is calculated based on the candidate region where the target instance is located and the corresponding query sample image, and the calculation includes:

[0030] The second optimization feature is subjected to an average pooling operation to obtain a second pooling feature;

[0031] The second pooling feature is subjected to a convolution operation to obtain a first convolution feature;

[0032] The region feature of the candidate region is obtained, and the region feature is subjected to a convolution operation to obtain a second convolution feature;

[0033] The first convolution feature is subtracted from the feature with the same channel identifier in the second convolution feature to obtain a second difference feature;

[0034] The second difference feature is subjected to a convolution operation to obtain the instance-level semantic similarity.

[0035] In an optional implementation, the pixel-level semantic similarity is calculated based on the candidate region where the target instance is located and the corresponding query sample image, and the calculation includes:

[0036] The second optimization feature is subjected to a convolution operation to obtain a third convolution feature;

[0037] The region feature of the candidate region is obtained, and the region feature is subjected to a convolution operation to obtain a fourth convolution feature;

[0038] An attention score is calculated based on the third convolution feature and the fourth convolution feature;

[0039] A product feature is obtained based on the product of the attention score and the second optimization feature;

[0040] differing the product feature from the region feature to obtain a third difference feature;

[0041] performing convolution operation on the third difference feature to obtain the pixel-level semantic similarity.

[0042] In an optional implementation, the similarity fusion based on the candidate feature of the candidate region, the image-level semantic similarity, the instance-level semantic similarity and the pixel-level semantic similarity to obtain a semantic similarity fusion result comprises:

[0043] performing convolution operation based on the image-level semantic similarity and the instance-level semantic similarity to obtain a first convolution similarity;

[0044] performing convolution operation based on the first convolution similarity, the candidate feature of the candidate region and the pixel-level semantic similarity to obtain a second convolution similarity;

[0045] determining the second convolution similarity as the semantic similarity fusion result.

[0046] A second aspect of the present application provides a small sample target detection device, the device comprises:

[0047] a sample acquisition module configured to acquire a plurality of query sample images and a target sample image corresponding to each of the query sample images, and to acquire a target instance belonging to the same category in each of the target sample images and the corresponding query sample image by constructing a small sample target detection model;

[0048] a first calculation module configured to calculate an image-level semantic similarity based on a candidate region where the target instance is located and the corresponding query sample image;

[0049] a second calculation module configured to calculate an instance-level semantic similarity based on the candidate region where the target instance is located and the corresponding query sample image;

[0050] a third calculation module configured to calculate a pixel-level semantic similarity based on the candidate region where the target instance is located and the corresponding query sample image;

[0051] a semantic fusion module configured to perform similarity fusion based on a candidate feature of the candidate region, the image-level semantic similarity, the instance-level semantic similarity and the pixel-level semantic similarity to obtain a semantic similarity fusion result;

[0052] a model training module configured to train the small sample target detection model based on the semantic similarity fusion result;

[0053] The category detection module is configured to detect, using the trained small sample target detection model, a region in a to-be-detected target image that is of the same category as an instance of the to-be-detected query image.

[0054] A third aspect of the present application provides an electronic device comprising a processor and a memory, wherein the processor is configured to implement the small sample target detection method when executing a computer program stored in the memory.

[0055] A fourth aspect of the present application provides a computer readable storage medium, wherein a computer program is stored on the computer readable storage medium, and the computer program is configured to implement the small sample target detection method when executed by a processor.

[0056] The present application calculates semantic similarities from multiple aspects, specifically, image-level, instance-level and pixel-level semantic similarities are calculated, the image-level semantic similarity corresponds to a coarse-grained object, the instance-level semantic similarity corresponds to a medium-grained object, and the pixel-level semantic similarity corresponds to a fine-grained object.

[0057] Secondly, the present application considers the fusion of semantic similarities of different levels. When fusing semantic similarities of different levels, the present application does not simply sum different semantic similarities by weighting, but fuses them through a series of convolution operations, so that the model can obtain more rich semantic information.

[0058] Furthermore, in order to better solve the problem of data imbalance in training data, the present application proposes a loss function capable of dynamically adjusting sample weights, to dynamically adjust the weights of positive and negative samples in the model training process, thereby solving the long-tail distribution problem of data in the training process, and balancing the learning effect of the model on positive and negative samples. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 is a flowchart of the small sample target detection method provided by the embodiment one of the present application.

[0060] Figure 2 is a schematic diagram of the feature extraction of the backbone network provided by the embodiment of the present application.

[0061] Figure 3 is a schematic diagram of the feature optimization provided by the embodiment of the present application.

[0062] Figure 4 is a schematic diagram of the candidate region generation provided by the embodiment of the present application.

[0063] Figure 5 is a schematic diagram of the image-level semantic similarity calculation provided by the embodiment of the present application.

[0064] Figure 6is a schematic diagram of example level semantic similarity calculation provided by an embodiment of the present application.

[0065] Figure 7 is a schematic diagram of pixel level semantic similarity calculation provided by an embodiment of the present application.

[0066] Figure 8 is a schematic diagram of semantic similarity fusion provided by an embodiment of the present application.

[0067] Figure 9 is a structural diagram of a small sample target detection device provided by the second embodiment of the present application.

[0068] Figure 10 is a structural schematic diagram of an electronic device provided by the third embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application will be described in detail below in combination with the drawings and specific embodiments. The embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0070] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used in the specification of the present application herein are only for the purpose of describing the embodiments in one optional embodiment, and are not intended to limit the present application.

[0071] The small sample target detection method provided by the embodiments of the present application is executed by an electronic device, and correspondingly, the small sample target detection device runs in the electronic device.

[0072] The embodiments of the present application can perform standardization processing on data based on artificial intelligence technology. Among them, artificial intelligence (AI) is to use digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results. Theory, method, technology and application system.

[0073] The basic technology of artificial intelligence generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc. Several major directions.

[0074] In view of the problem that the model can successfully detect the object category without training unknown data, the prior art proposes a small sample target detection method. Small sample target detection is to give a query sample (unknown category) and a target sample (to be detected image), and to detect and frame the objects in the target sample of the same category as the query sample. The core of small sample target detection includes the following two points: one is to extract the features of the query sample and the target sample; the second is to calculate the semantic similarity between the features of the query sample and the target sample.

[0075] The existing calculation of the semantic similarity between the features of the query sample and the target sample is that some regard the query sample feature and the target sample feature as word vectors, and regard the cosine similarity between the word vectors as the semantic similarity result, and some regard the attention score between the query sample feature and the target sample feature as the semantic similarity result. However, the above method only regards the feature as a whole and calculates the similarity between the whole, without considering the influence of the internal details of the sample feature on the semantic similarity, resulting in poor processing effect of the model on the problems such as object occlusion and object self-rotation. Secondly, in the sampling process, only the samples consistent with the query sample category are considered as positive samples, and the rest are considered as negative samples, which leads to an imbalance between positive and negative samples, thereby reducing the learning effect of the model on positive samples and further reducing the generalization ability of the model on unknown categories.

[0076] Embodiment one

[0077] Figure 1 is the flowchart of the small sample target detection method provided by the embodiment one of the application. The small sample target detection method specifically includes the following steps, and the order of the steps in the flowchart can be changed, and some can be omitted according to different needs.

[0078] S11, acquiring a plurality of query sample images and a target sample image corresponding to each query sample image, and acquiring a target instance belonging to the same category in each target sample image and the corresponding query sample image through a small sample target detection model.

[0079] The electronic device first constructs a query-target sample image pair as training data, then constructs a network architecture of the small sample target detection model, and then trains the small sample target detection model through the training data to obtain a trained small sample target detection model. The trained small sample target detection model can detect the target instance belonging to the same category in the to-be-detected target image and the to-be-detected query image.

[0080] The query-target sample image pair refers to a query sample image and a target sample image including instances of the same category. The construction process of the query-target sample image pair can be: randomly selecting an image, obtaining the position information of the instance in the image, cropping the randomly selected image according to the position information of the instance, taking the cropped image as the query sample image, randomly selecting another image containing instances of the same category as the instance in the query sample image as the target sample image, and taking the query sample image and the corresponding target sample image as the query-target sample image pair. The first category label of the instance in the query sample image is labeled, and the second category label of each instance in the target sample image is labeled.

[0081] It should be noted that the query sample image includes a single and complete instance, that is, there is only one category of instance in the query sample image. The target sample image can include instances of multiple categories, and at least one instance of the multiple categories has the same category as the instance in the query sample image. For example, there is only a kitten in the query sample image, and the target sample image can include not only a kitten but also a dog, a house, etc. The first category label of the instance in the query sample image is labeled as “cat”, and the second category label of the instance in the target sample image is labeled as “cat”, “dog”, and “house”. The electronic device can exhaust all categories in the target sample image and set a number for each category as the category label of the category. For example, the number “1” is set for the category of the dog as the category label of the dog, and the number “2” is set for the category of the kitten as the category label of the kitten.

[0082] The query-target sample image pair is input into the small sample target detection model, and the output of the small sample target detection model is a candidate instance in the target sample image that has the same category as the instance in the query sample image or a similar category and a candidate region corresponding to the candidate instance. There can be multiple candidate regions, and each candidate region corresponds to a detection box.

[0083] In an optional embodiment, the electronic device can construct a small sample target detection model based on the R-CNN framework, which includes a feature extraction network, a feature optimization network connected to the feature extraction network, and a candidate region generation network connected to the feature optimization network, and the candidate region generation network is connected to the R-CNN network. The full name of R-CNN is Region-CNN, which is the first algorithm that successfully applies deep learning to target detection. R-CNN is based on convolutional neural networks, linear regression, and support vector machines, and realizes target detection technology.

[0084] In an optional embodiment, the small sample target detection model includes:

[0085] The feature extraction network is used to extract first sample features of the target sample images and second sample features of the query sample images.

[0086] The feature optimization network is used to optimize the first sample features to obtain first optimized features and to optimize the second sample features to obtain second optimized features.

[0087] The candidate region generation network is used to obtain target instances belonging to the same category as instances in the corresponding query sample images in each of the target sample images based on the first optimized features and the second optimized features.

[0088] ResNet-50 can be used as a backbone of the small sample target detection model, and the ResNet-50 network can extract features with strong expression ability. As shown in FIG. 2, a target sample image is input into the backbone network for feature extraction to obtain first sample features of the target sample image, and a query sample image is input into the backbone network for feature extraction to obtain second sample features of the query sample image. Figure 2

[0089] Feature Pyramid Networks (FPN) can be used as a neck of the small sample target detection model to optimize features output by the backbone. The neck is located between the backbone and the head, and the head is a network that obtains network output content. The pyramid network can up-sample the features at the bottom layer and fuse the features at the bottom layer to obtain high-resolution and strong semantic features, thereby optimizing the features. As shown in FIG. 3, the first sample features are input into the feature pyramid network for feature optimization, and the first optimized features are output by the feature pyramid network. The second sample features are input into the feature pyramid network for feature optimization, and the second optimized features are output by the feature pyramid network. Figure 3

[0090] Region Proposal Network (RPN) is used to learn and generate candidate boxes containing foreground instances from features. As shown in FIG. 4, the candidate region generation network obtains candidate regions of candidate instances belonging to the same category as instances in the query sample images in the target sample images based on the first optimized features and the second optimized features. Figure 4

[0091] S12, based on the candidate region where the target instance is located and the corresponding query sample image, an image-level semantic similarity is calculated.

[0092] ​​​The small sample object detection model also includes a semantic similarity calculation model, which inputs the features of each candidate region in the target sample image and the features of the query sample image into the semantic similarity calculation model to obtain the semantic similarity between each candidate region and the query sample image.

[0093] The semantic similarity calculation model includes four sub-models: a first sub-model, a second sub-model, a third sub-model, and a fourth sub-model. The first sub-model calculates image-level semantic similarity, the second sub-model calculates instance-level semantic similarity, the third sub-model calculates pixel-level semantic similarity, and the fourth sub-model fuses the semantic similarity results from the first, second, and third sub-models.

[0094] In an optional implementation, the step of calculating the image-level semantic similarity based on the candidate region where the target instance is located and the corresponding query sample image includes:

[0095] Obtain the regional features of the candidate region;

[0096] The second optimized feature of the query sample image is subjected to average pooling to obtain the first pooled feature;

[0097] The first difference feature is obtained by subtracting the first pooling feature from the features with the same channel identifier in the region features;

[0098] The first difference feature is convolved to obtain the image-level semantic similarity.

[0099] When the candidate region generation network outputs candidate regions of candidate instances in the target sample image that are of the same category as instances in the query sample image, it can also output the features of the candidate instances as candidate features of the candidate regions.

[0100] like Figure 5 As shown, assuming the candidate feature dimension of the candidate region is C*K*K, and the dimension of the second optimized feature of the query sample image is C*K*K, where C is the number of channels and K*K is the width*height of the candidate region, the second optimized feature is input into the average pooling layer of the first computational sub-model for average pooling operation to obtain the first pooled feature, which has a dimension of C*1*1. Average pooling operation refers to averaging the features of each channel.

[0101] The features of each channel of the first pooling feature are copied and tiled onto the corresponding channel of the candidate feature and the difference is calculated to obtain the first difference feature, which has a dimension of C*K*K.

[0102] Finally, a 1*1 convolution kernel is used to perform a convolution operation on the first differential feature, resulting in a feature with a dimension of C / 2*K*K.

[0103] Since the 1x1 convolution operation targets the regional features of the candidate region and the second optimized features of the query sample image, the features obtained from the convolution operation are determined as the image-level semantic similarity. The number of channels for the image-level semantic similarity is half the dimension of the candidate features.

[0104] S13, calculate the instance-level semantic similarity based on the candidate region where the target instance is located and the corresponding query sample image.

[0105] The electronic device simultaneously inputs the regional features of the candidate region and the second optimized features of the query sample image into the second calculation sub-model of the semantic similarity calculation model, and calculates the instance-level semantic similarity through the second calculation sub-model.

[0106] In an optional implementation, calculating the instance-level semantic similarity based on the candidate region where the target instance is located and the corresponding query sample image includes:

[0107] The second optimized feature is subjected to average pooling to obtain the second pooled feature;

[0108] The first convolutional feature is obtained by performing a convolution operation on the second pooling feature;

[0109] Obtain the regional features of the candidate region, and perform a convolution operation on the regional features to obtain the second convolutional features;

[0110] The second difference feature is obtained by subtracting the features with the same channel identifier from the first convolutional feature and the second convolutional feature.

[0111] Perform a convolution operation on the second difference feature to obtain the instance-level semantic similarity.

[0112] like Figure 6 As shown, assuming the dimension of the candidate feature of the candidate region is C*K*K, and the dimension of the second optimized feature of the query sample image is C*K*K, where C is the number of channels and K*K is the width*height of the candidate region, the second optimized feature is input into the average pooling layer of the second calculation sub-model for average pooling operation to obtain the second pooling feature, and the dimension of the second pooling feature is C*1*1.

[0113] Then, the second pooling feature is separated by channels to obtain a plurality of first channel features, and the candidate feature is separated by channels to obtain a plurality of second channel features, wherein the dimension of each first channel feature and each second channel feature is C*1*1.

[0114] Next, a deep convolution operation is performed on the second pooling feature to obtain a first convolution feature, and a deep convolution operation is performed on the region feature to obtain a second convolution feature; the dimensions of the first convolution feature and the second convolution feature are both C*K*K.

[0115] Each channel of the first convolution feature is copied and tiled onto the corresponding channel of the second convolution feature and is subtracted to obtain a second difference feature, and the dimension of the second difference feature is C*K*K.

[0116] Finally, a 1*1 convolution kernel is used to perform a convolution operation on the second difference feature, and the dimension of the obtained feature is C / 2*K*K.

[0117] Since the 1*1 convolution operation is performed on the convolution feature obtained by performing a convolution operation on the region feature and the convolution feature obtained by performing a convolution operation on the second optimization feature, the feature obtained by the 1*1 convolution operation is determined as the instance-level semantic similarity. The number of channels of the instance-level semantic similarity is half of the dimension of the candidate feature.

[0118] S14, pixel-level semantic similarity is calculated based on the candidate region where the target instance is located and the corresponding query sample image.

[0119] The electronic device inputs the region feature of the candidate region and the second optimization feature of the query sample image to a third calculation sub-model of the semantic similarity calculation model at the same time, and calculates the pixel-level semantic similarity through the third calculation sub-model.

[0120] In an optional implementation, the pixel-level semantic similarity is calculated based on the candidate region where the target instance is located and the corresponding query sample image, which includes:

[0121] A convolution operation is performed on the second optimization feature to obtain a third convolution feature;

[0122] The region feature of the candidate region is obtained, and a convolution operation is performed on the region feature to obtain a fourth convolution feature;

[0123] An attention score is calculated based on the third convolution feature and the fourth convolution feature;

[0124] A product feature is obtained based on the product of the attention score and the second optimization feature;

[0125] S14, performing a difference operation between the product feature and the region feature to obtain a third difference feature.

[0126] S13, performing a convolution operation on the third difference feature to obtain the pixel-level semantic similarity.

[0127] As shown in Figure 7 the candidate feature of the candidate region has a dimension of C*K*K, and the second optimization feature of the query sample image has a dimension of C*K*K, where C is a channel number, and K*K is a width*height of the candidate region. First, a 1*1 convolution kernel is used to perform a convolution operation on the second optimization feature to obtain a third convolution feature, and a 1*1 convolution kernel is used to perform a convolution operation on the region feature to obtain a fourth convolution feature. The third convolution feature and the fourth convolution feature have a dimension of C / 8*K*K.

[0128] Then, the third convolution feature and the fourth convolution feature are input into a softmax layer for prediction to obtain an attention score as an attention weight. The attention weight has a dimension of C / 8*K*K.

[0129] A product feature is obtained based on a product of the attention weight and the second optimization feature. Then, a difference operation is performed between the obtained product feature and the region feature, and an absolute value of the difference is obtained to obtain a third difference feature. The third difference feature has a dimension of C*K*K.

[0130] Finally, a 1*1 convolution kernel is used to perform a convolution operation on the third difference feature to obtain a feature with a dimension of C / 2*K*K.

[0131] Since the last 1*1 convolution operation is performed on the product feature, the feature obtained by the last 1*1 convolution operation is determined as the pixel-level semantic similarity. The channel number of the pixel-level semantic similarity is half of the dimension of the candidate feature.

[0132] S15, based on the candidate feature of the candidate region, the image-level semantic similarity, the instance-level semantic similarity, and the pixel-level semantic similarity, performing similarity fusion to obtain a semantic similarity fusion result.

[0133] After the electronic device obtains the image-level semantic feature, the instance-level semantic feature, and the pixel-level semantic feature, the electronic device inputs the candidate feature of the candidate region, the image-level semantic feature, the instance-level semantic feature, and the pixel-level semantic feature into a fourth calculation sub-model of the semantic similarity calculation model. The fourth calculation sub-model is used to perform semantic similarity fusion to obtain a semantic similarity fusion result.

[0134] The fourth calculation sub-model can include a feature alignment network, which is configured by using a learnable full connection layer to learn the fusion relationship among the candidate features of the candidate region, the image-level semantic features, the instance-level semantic features, and the pixel-level semantic features.

[0135] In an optional implementation, the similarity fusion based on the candidate features of the candidate region, the image-level semantic similarity, the instance-level semantic similarity, and the pixel-level semantic similarity to obtain the semantic similarity fusion result includes:

[0136] performing convolution operation based on the image-level semantic similarity and the instance-level semantic similarity to obtain a first convolution similarity;

[0137] performing convolution operation based on the first convolution similarity, the candidate features of the candidate region, and the pixel-level semantic similarity to obtain a second convolution similarity;

[0138] determining the second convolution similarity as the semantic similarity fusion result.

[0139] As shown in Figure 8 the image-level semantic similarity and the instance-level semantic similarity are spliced to obtain first spliced features, and then the first spliced features are input into a first convolution layer of the fourth calculation sub-model to perform convolution operation, and the first convolution layer of the fourth calculation sub-model uses a 1*1 convolution kernel.

[0140] Then, the candidate features of the candidate region and the pixel-level semantic similarity are spliced to obtain second spliced features. Then, the first convolution similarity and the second spliced features are spliced to obtain third spliced features.

[0141] Finally, the third spliced features are input into a second convolution layer of the fourth calculation sub-model to perform convolution operation, and the second convolution layer of the fourth calculation sub-model uses a 1*1 convolution kernel. The second convolution layer outputs a second convolution similarity, and the dimension is 3C / 2*K*K.

[0142] S16, training the small sample target detection model based on the semantic similarity fusion result.

[0143] After obtaining the semantic similarity fusion result, the electronic device inputs the semantic similarity fusion result into the R-CNN, calculates a loss value by using a loss function, and trains the small sample target detection model by using a back propagation algorithm.

[0144] In an optional implementation, the training of the small sample target detection model based on the semantic similarity fusion result includes:

[0145] outputting, by the few-shot object detection model, a predicted class label of the target instance and a predicted bounding box of the target instance in each of the target sample images;

[0146] dividing, based on the predicted bounding box, the target sample images into positive sample images and negative sample images;

[0147] for the positive sample images, calculating a first cross-entropy loss value based on the predicted class label and a true class label of the instance by using a preset first loss function;

[0148] for the negative sample images, calculating a second cross-entropy loss value based on the predicted class label and the true class label of the instance by using a preset second loss function;

[0149] calculating a total loss value based on the first cross-entropy loss value and the second cross-entropy loss value;

[0150] training, by using a gradient descent algorithm, the few-shot object detection model based on the loss value to obtain a trained few-shot object detection model.

[0151] Each of the target instances in each of the target sample images corresponds to a true bounding box and a true class label, and each of the target instances in each of the target sample images also corresponds to a predicted bounding box and a predicted class label. The true bounding box and the true class label are manually processed by a human, and the predicted bounding box and the predicted class label are output by the few-shot object detection model.

[0152] For each of the target instances in each of the target sample images, the intersection over union of the true bounding box and the predicted bounding box is calculated, and it is determined whether the intersection over union is greater than a preset intersection over union threshold (for example, 50%) to determine whether the target sample image is a positive sample image or a negative sample image. When the intersection over union of the true bounding box and the predicted bounding box is greater than the preset intersection over union threshold, it is determined that the target sample image is a positive sample image. When the intersection over union of the true bounding box and the predicted bounding box is not greater than the preset intersection over union threshold, it is determined that the target sample image is a negative sample image.

[0153] The calculation process of the intersection over union of the true bounding box and the predicted bounding box includes: calculating a first area of the union of the true bounding box and the predicted bounding box, calculating a second area of the intersection of the true bounding box and the predicted bounding box, and calculating a ratio of the second area to the first area as the intersection over union of the true bounding box and the predicted bounding box.

[0154] The above embodiment calculates the intersection over union of the real box and the predicted box, and compares the calculated intersection over union with a preset intersection over union threshold, to determine the prediction accuracy of the small sample prediction model on the position of the target instance in the target sample image. When the intersection over union of the real box and the predicted box is greater than the preset intersection over union threshold, it indicates that the position of the predicted box is close to the position of the real box, thereby indicating that the small sample prediction model is more accurate in predicting the position of the target instance in the target sample image. In other words, the target sample image can play a positive role in training the small sample target detection model, and therefore, the target sample image can be identified as a positive sample image. When the intersection over union of the real box and the predicted box is not greater than the preset intersection over union threshold, it indicates that the position of the predicted box is far from the position of the real box, thereby indicating that the small sample prediction model is not accurate in predicting the position of the target instance in the target sample image. In other words, the target sample image plays a negative role in training the small sample target detection model, and therefore, the target sample image can be identified as a negative sample image.

[0155] wherein the total loss value is represented as follows:

[0156]

[0157]

[0158]

[0159] wherein LOSS is the total loss value, N is the number of target sample images, parameter λ is a network parameter to be learned, N p represents the number of positive sample images in the target sample images, N fp represents the number of false positive sample images, N tn represents the number of true negative sample images, represents the cross-entropy loss of the i-th target sample image, i = 1, 2, …, N. The cross-entropy loss of the i-th target sample image is calculated as follows: wherein Y represents the real class label of the i-th target sample image, Y ′ represents the predicted class label of the i-th target sample image output by the small sample target detection model.

[0160] The false positive sample image refers to a negative sample image being predicted as a positive sample image, and the true negative sample image refers to a negative sample image being predicted as a negative sample image. After the plurality of target sample images are divided into positive sample images and negative sample images based on the prediction frame, whether the sample image is a false positive sample image or a true negative sample image can be determined by the prediction class label output by the small sample target detection model and the true class label. Specifically, for the negative sample image determined based on the prediction frame, if the prediction class label output by the small sample target detection model is consistent with the true class label, the negative sample image is a false positive sample image. For the negative sample image determined based on the prediction frame, if the prediction class label output by the small sample target detection model is inconsistent with the true class label, the negative sample image is a true positive sample image.

[0161] By dynamically adjusting the corresponding weights of the positive sample images and the negative sample images in the loss function, the learning effect of the model on different samples can be adjusted, and the generalization ability of the model can be improved.

[0162] S17, using the trained small sample target detection model to detect the region in the to-be-detected target image that belongs to the same category as the instance of the to-be-detected query image.

[0163] The essential idea of the small sample target detection model is to learn the similarity measurement between each candidate frame in the target image and the query image in the visual domain space. This type of model only focuses on whether the visual domain features of the foreground instance in the candidate frame and the visual domain features of the query image are similar, and does not focus on which category the detected target belongs to, and is independent of the category. That is, during training of the small sample target detection model, the class labels of the query sample image and the target sample image are known, and the purpose of training is to learn the similarity measurement between the query sample image and the target sample image, which is independent of the category. When the trained small sample target detection model is tested, the learned similarity measurement is directly used on the to-be-detected target image.

[0164] The to-be-detected target image and the to-be-detected query image are constructed into a target-query image pair, input into the trained small sample target detection model, and the detection result of the to-be-detected target image is obtained, that is, the instance belonging to the same category as the to-be-detected query image is found in the to-be-detected target image, and the detection task is completed.

[0165] Due to the problems such as occlusion, deformation, translation and rotation of objects in the target sample, in order to better find the position information of the query sample in the target sample, the semantic similarity is calculated from multiple aspects, specifically, the semantic similarity at the image level, the instance level and the pixel level is calculated, the semantic similarity at the image level corresponds to the detection of coarse-grained objects, the semantic similarity at the instance level corresponds to the detection of medium-grained objects, and the semantic similarity at the pixel level corresponds to the detection of fine-grained objects.

[0166] Secondly, the application considers the fusion problem of different levels of semantic similarity. When fusing different levels of semantic similarity, the application does not simply weight and sum different semantic similarities, but fuses them through a series of convolution operations, so that the model can obtain more rich semantic information.

[0167] Furthermore, in order to better solve the problem of data imbalance in training data, the application proposes a loss function capable of dynamically adjusting sample weights, to dynamically adjust the weights of positive and negative samples in the model training process, thereby solving the long-tail distribution problem of data in the training process, and balancing the learning effect of the model on positive and negative samples.

[0168] Embodiment Two

[0169] Figure 9 is the structural diagram of the small sample target detection device provided by the second embodiment of the application.

[0170] In some embodiments, the small sample target detection device 90 can include a plurality of functional modules composed of computer program segments. The computer programs of each program segment in the small sample target detection device 90 can be stored in the memory of the electronic device and executed by at least one processor to perform the functions of small sample target detection (see Figure 1 Description).

[0171] In this embodiment, the small sample target detection device 90 can be divided into a plurality of functional modules according to the functions it performs. The functional modules can include a sample acquisition module 901, a first calculation module 902, a second calculation module 903, a third calculation module 904, a semantic fusion module 905, a model training module 906, and a category detection module 907. The module referred to by the application refers to a series of computer program segments that can be executed by at least one processor and can complete a fixed function, which are stored in the memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0172] The sample acquisition module 901 is configured to acquire a plurality of query sample images and a target sample image corresponding to each query sample image, and acquire target instances belonging to the same category in each target sample image and the corresponding query sample image through a small sample target detection model.

[0173] The first calculation module 902 is configured to calculate an image-level semantic similarity based on a candidate region where the target instance is located and the corresponding query sample image.

[0174] The second calculation module 903 is configured to calculate an instance-level semantic similarity based on the candidate region where the target instance is located and the corresponding query sample image.

[0175] The third calculation module 904 is configured to calculate a pixel-level semantic similarity based on the candidate region where the target instance is located and the corresponding query sample image.

[0176] The semantic fusion module 905 is configured to perform similarity fusion based on the candidate feature of the candidate region, the image-level semantic similarity, the instance-level semantic similarity, and the pixel-level semantic similarity to obtain a semantic similarity fusion result.

[0177] The model training module 906 is configured to train the small sample target detection model based on the semantic similarity fusion result.

[0178] The category detection module 907 is configured to detect a region in a to-be-detected target image that is of the same category as the instance in the to-be-detected query image by using the small sample target detection model after training.

[0179] Embodiment Three

[0180] The embodiment provides a computer readable storage medium, and the computer readable storage medium stores a computer program. When the computer program is executed by a processor, steps in the small sample target detection method embodiments described above are implemented, for example Figure 1 S11-S17 shown in the figure:

[0181] S11, a plurality of query sample images and a target sample image corresponding to each query sample image are obtained, and a small sample target detection model is used to obtain a target instance in each target sample image that belongs to the same category as the corresponding query sample image;

[0182] S12, an image-level semantic similarity is calculated based on the candidate region where the target instance is located and the corresponding query sample image;

[0183] S13, an instance-level semantic similarity is calculated based on the candidate region where the target instance is located and the corresponding query sample image;

[0184] S14, a pixel-level semantic similarity is calculated based on the candidate region where the target instance is located and the corresponding query sample image;

[0185] S15, similarity fusion is performed based on the candidate feature of the candidate region, the image-level semantic similarity, the instance-level semantic similarity, and the pixel-level semantic similarity to obtain a semantic similarity fusion result;

[0186] S16, training the small sample target detection model based on the semantic similarity fusion result;

[0187] S17, detecting a region in a to-be-detected target image that is of the same category as an instance of a to-be-detected query image using the trained small sample target detection model.

[0188] Alternatively, the computer program is executed by a processor to realize the functions of each module / unit in the above-mentioned apparatus embodiment, for example Figure 9 the modules 901-907 in the apparatus 1000:

[0189] The sample acquisition module 901 is configured to acquire a plurality of query sample images and a target sample image corresponding to each query sample image, and acquire a target instance belonging to the same category as the corresponding query sample image in each target sample image by constructing a small sample target detection model.

[0190] The first calculation module 902 is configured to calculate an image-level semantic similarity based on a candidate region where the target instance is located and the corresponding query sample image.

[0191] The second calculation module 903 is configured to calculate an instance-level semantic similarity based on the candidate region where the target instance is located and the corresponding query sample image.

[0192] The third calculation module 904 is configured to calculate a pixel-level semantic similarity based on the candidate region where the target instance is located and the corresponding query sample image.

[0193] The semantic fusion module 905 is configured to perform similarity fusion based on the candidate features of the candidate region, the image-level semantic similarity, the instance-level semantic similarity, and the pixel-level semantic similarity to obtain a semantic similarity fusion result.

[0194] The model training module 906 is configured to train the small sample target detection model based on the semantic similarity fusion result.

[0195] The category detection module 907 is configured to detect a region in a to-be-detected target image that is of the same category as an instance of a to-be-detected query image using the trained small sample target detection model.

[0196] Embodiment Four

[0197] Referring to Figure 10 FIG. 1 shows a structural schematic diagram of an electronic device provided in Embodiment Three of the present application. In the preferred embodiment of the present application, the electronic device 10 comprises a memory 101, at least one processor 102, at least one communication bus 103, and a transceiver 104.

[0198] Those skilled in the art should understand that, Figure 10 The structure of the electronic device shown does not constitute a limitation of the embodiments of the present application, and can be a bus structure or a star structure. The electronic device 10 can also include more or fewer other hardware or software, or different component arrangements than those shown.

[0199] In some embodiments, the electronic device 10 is a device capable of automatically performing numerical calculation and / or information processing according to pre-set or stored instructions. The hardware of the electronic device 10 includes, but is not limited to, a microprocessor, an application-specific integrated circuit, a programmable gate array, a digital processor, and an embedded device. The electronic device 10 can also include a client device, which includes, but is not limited to, any electronic product that can interact with a client through a keyboard, a mouse, a remote control, a touchpad, or a voice control device, such as a personal computer, a tablet computer, a smart phone, a digital camera, and the like.

[0200] The electronic device 10 is only an example. Other existing or future electronic products, such as those that can be adapted to the present application, should also be included within the scope of the present application and are hereby incorporated by reference.

[0201] In some embodiments, the memory 101 stores a computer program that, when executed by the at least one processor 102, implements all or part of the steps of the small sample target detection method as described. The memory 101 includes a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk memory, a magnetic disk memory, a magnetic tape memory, or any other computer-readable medium capable of carrying or storing data.

[0202] Further, the computer readable storage medium can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs required by at least one function, and the like; and the data storage area can store data created according to the use of the blockchain node, and the like.

[0203] The blockchain referred to in the present application is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm. The blockchain is essentially a decentralized database, and is a series of data blocks associated using cryptographic methods, each of which contains information of a batch of network transactions, for verifying the validity (anti-fake) of the information and generating the next block. The blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer, and the like.

[0204] In some embodiments, the at least one processor 102 is a control unit of the electronic device 10, which connects various components of the entire electronic device 10 by using various interfaces and lines, and performs various functions of the electronic device 10 and processes data by running or executing programs or modules stored in the memory 101 and calling data stored in the memory 101. For example, the at least one processor 102 implements all or part of the steps of the small sample target detection method described in the embodiments of the present application when executing the computer program stored in the memory, or implements all or part of the functions of the small sample target detection device. The at least one processor 102 can be composed of integrated circuits, for example, can be composed of a single packaged integrated circuit, or can be composed of multiple packaged integrated circuits with the same function or different functions, including one or more central processing units (CPU), microprocessors, digital processing chips, graphics processors, and combinations of various control chips, etc.

[0205] In some embodiments, the at least one communication bus 103 is arranged to realize the connection and communication between the memory 101, the at least one processor 102, and the like.

[0206] Although not shown, the electronic device 10 can further include a power supply (such as a battery) for powering various components. Preferably, the power supply can be logically connected to the at least one processor 102 through a power management device, so as to realize the functions of managing charging, discharging, and power consumption management, and the like through the power management device. The power supply can also include one or more direct current or alternating current power supplies, recharging devices, power supply fault detection circuits, power supply converters or inverters, power supply status indicators, and the like. The electronic device 10 can further include various sensors, Bluetooth modules, Wi-Fi modules, and the like, which are not described here again.

[0207] The integrated units in the form of software function modules can be stored in a computer readable storage medium. The software function modules are stored in a storage medium, and include a plurality of instructions for causing a computer device (which can be a personal computer, an electronic device, or a network device, etc.) or a processor to execute part of the method described in various embodiments of the present application.

[0208] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other manners. For example, the described apparatus embodiments are merely schematic. For example, the division of the modules is merely a logical function division. There can be another division manner for the actual implementation.

[0209] The modules illustrated as separated components can or can not be physically separated, and the components illustrated as modules can or can not be physical units. They can be located in one position or distributed on multiple network units. Some or all of the modules can be selected according to the actual needs to achieve the purposes of the embodiments.

[0210] In addition, each function module in the various embodiments of the present application can be integrated in a processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware, or in the form of hardware plus software function module.

[0211] It is obvious for those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered as exemplary and non-limiting, and the scope of the present application is defined by the appended claims rather than the above description, and all changes falling within the meaning and range of equivalent elements of the claims are intended to be included in the present application. Any reference signs in the claims should not be considered as limiting the claims. In addition, it is obvious that the word "comprise" does not exclude other units or, the singular does not exclude the plural. The plurality of units or devices stated in the description can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not represent any particular order.

[0212] Finally, it should be noted that the above examples are merely intended to illustrate the technical solutions of the present application and not to limit the present application. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. A small sample target detection method, characterized in that, The method comprises: obtaining a plurality of query sample images and a target sample image corresponding to each of the query sample images, and obtaining, by constructing a small sample target detection model, a target instance belonging to the same category in each of the target sample images and the corresponding query sample image; calculating an image-level semantic similarity based on a candidate region where the target instance is located and the corresponding query sample image; calculating an instance-level semantic similarity based on the candidate region where the target instance is located and the corresponding query sample image; calculating a pixel-level semantic similarity based on the candidate region where the target instance is located and the corresponding query sample image; performing similarity fusion based on candidate features of the candidate region, the image-level semantic similarity, the instance-level semantic similarity, and the pixel-level semantic similarity to obtain a semantic similarity fusion result, including: performing convolution operation based on the image-level semantic similarity and the instance-level semantic similarity to obtain a first convolution similarity; performing convolution operation based on the first convolution similarity, the candidate features of the candidate region, and the pixel-level semantic similarity to obtain a second convolution similarity; and determining the second convolution similarity as the semantic similarity fusion result; training the small sample target detection model based on the semantic similarity fusion result; detecting, using the trained small sample target detection model, a region in a to-be-detected target image that belongs to the same category as an instance of a to-be-detected query image.

2. The small sample target detection method of claim 1, wherein, The training of the small sample target detection model based on the semantic similarity fusion result comprises: outputting, by the small sample target detection model, a predicted category label of the target instance in each of the target sample images and a predicted box of the target instance, the predicted box corresponding to the candidate region; dividing a plurality of the target sample images into positive sample images and negative sample images based on the predicted box; for the positive sample images, calculating a first cross-entropy loss value based on the predicted category label and a real category label of the instance using a preset first loss function; for the negative sample images, calculating a second cross-entropy loss value based on the predicted category label and the real category label of the instance using a preset second loss function; calculating a total loss value based on the first cross-entropy loss value and the second cross-entropy loss value; training the small sample target detection model based on the loss value using a gradient descent algorithm to obtain a trained small sample target detection model.

3. The small sample target detection method of claim 1, wherein, The construction of the small sample target detection model to obtain a target instance belonging to the same category in each of the target sample images and the corresponding query sample image comprises: constructing a small sample target detection model with R-CNN as the main framework, the small sample target detection model comprising a feature extraction network, a feature optimization network, and a candidate region generation network; extracting, by the feature extraction network, first sample features of the target sample images and second sample features of the query sample images; The first sample feature is optimized by the feature optimization network to obtain a first optimized feature, and the second sample feature is optimized to obtain a second optimized feature; The candidate region generation network is used to obtain, based on the first optimized feature and the second optimized feature, a target instance belonging to the same category in each of the target sample images and the corresponding query sample images.

4. The small sample target detection method of claim 3, wherein, The image-level semantic similarity is calculated based on the candidate region where the target instance is located and the corresponding query sample image, and includes: A region feature of the candidate region is obtained; The second optimized feature of the query sample image is subjected to an average pooling operation to obtain a first pooled feature; The first pooled feature is subtracted from the feature with the same channel identifier in the region feature to obtain a first difference feature; The first difference feature is subjected to a convolution operation to obtain the image-level semantic similarity.

5. The small sample target detection method of claim 3, wherein, The instance-level semantic similarity is calculated based on the candidate region where the target instance is located and the corresponding query sample image, and includes: The second optimized feature is subjected to an average pooling operation to obtain a second pooled feature; The second pooled feature is subjected to a convolution operation to obtain a first convolution feature; A region feature of the candidate region is obtained, and the region feature is subjected to a convolution operation to obtain a second convolution feature; The first convolution feature is subtracted from the feature with the same channel identifier in the second convolution feature to obtain a second difference feature; The second difference feature is subjected to a convolution operation to obtain the instance-level semantic similarity.

6. The small sample target detection method of claim 3, wherein, The pixel-level semantic similarity is calculated based on the candidate region where the target instance is located and the corresponding query sample image, and includes: The second optimized feature is subjected to a convolution operation to obtain a third convolution feature; A region feature of the candidate region is obtained, and the region feature is subjected to a convolution operation to obtain a fourth convolution feature; An attention score is calculated based on the third convolution feature and the fourth convolution feature; A product feature is obtained based on the product of the attention score and the second optimized feature; The product feature is subtracted from the region feature to obtain a third difference feature; The third difference feature is subjected to a convolution operation to obtain the pixel-level semantic similarity.

7. A small sample target detection device, characterized in that, The device includes: A sample acquisition module is configured to acquire a plurality of query sample images and a target sample image corresponding to each of the query sample images, and to obtain, by constructing a small sample target detection model, a target instance belonging to the same category in each of the target sample images and the corresponding query sample images; A first calculation module is configured to calculate an image-level semantic similarity based on a candidate region where the target instance is located and a corresponding query sample image; A second calculation module is configured to calculate an instance-level semantic similarity based on a candidate region where the target instance is located and a corresponding query sample image; A third calculation module is configured to calculate a pixel-level semantic similarity based on a candidate region where the target instance is located and a corresponding query sample image. The semantic fusion module is configured to perform similarity fusion based on the candidate feature of the candidate region, the image-level semantic similarity, the instance-level semantic similarity, and the pixel-level semantic similarity to obtain a semantic similarity fusion result, including: performing convolution operation based on the image-level semantic similarity and the instance-level semantic similarity to obtain a first convolution similarity; performing convolution operation based on the first convolution similarity, the candidate feature of the candidate region, and the pixel-level semantic similarity to obtain a second convolution similarity; and determining the second convolution similarity as the semantic similarity fusion result; The model training module is configured to train the small sample target detection model based on the semantic similarity fusion result. The category detection module is configured to use the trained small sample target detection model to detect a region in a to-be-detected target image that is of the same category as the instance of the to-be-detected query image.

8. An electronic device, comprising: The electronic device includes a processor and a memory, and the processor is configured to implement the small sample target detection method of any one of claims 1 to 6 when executing a computer program stored in the memory.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the small sample target detection method of any one of claims 1 to 6.

Images