A small sample image detection model training method and an image detection method

Abstract

The application discloses a kind of small sample image detection model training method and image detection method, belong to deep learning technical field, the base class training method includes: using support image set, the labeled query image and each target class in class trains original image detection model until total loss reaches convergence, obtains base class image detection model;Support feature and the semantic embedding of target class are input into variational autoencoder, compared with the support feature of specific class and reconstruction feature, latent vector in VAE intermediate process contains more general features of class prototype;From distribution, latent vector with containing semantic information is sampled and is aggregated with query feature, so that the estimated new class center is closer to the real class center;In addition, reconstruction loss in VAE, KL divergence, classification loss, regression loss and semantic consistency loss constitute total loss, which can greatly improve the training precision.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of deep learning, and more particularly relates to a small sample image detection model training method and an image detection method. BACKGROUND

[0002] The directed network image retrieval is mainly used for directed screening and extraction of images containing specific target objects from massive images. However, in the face of massive network images, only a small part of the images can be labeled, and such small sample phenomenon is more typical for new things or non-hot events on the Internet. In recent years, the target detection technology based on deep learning has been widely studied, and the target detection aims to identify and accurately locate the target in the image, which usually needs a large amount of labeled data for training.

[0003] In actual scenarios, it is difficult and costly to obtain large-scale labeled samples, therefore, the small sample target detection (Few-shot Object Detection, FSOD) method under the condition of data scarcity has also attracted more and more attention. The main idea of FSOD is to train a base class detection model on a base class dataset with a large amount of labeled information, and then use a small amount of labeled new class dataset and the prior knowledge provided by the base class model to realize the detection of the new class. Through FSOD, the model can effectively filter out the images containing target objects from a large number of unlabeled images under the condition of only a small amount of labeled images. At present, the meta-learning method in the paradigm of small sample target detection trains the model by simulating multiple small sample tasks, so that the model can quickly adapt to the detection task of new classes. Meta-learning obtains task-level knowledge pre-trained on the base class through scenario training, and enhances the generalization ability of the model when encountering new classes.

[0004] However, since each class has only K samples, the meta-learning method highly depends on the quality of the support set, and the traditional meta-learning model regards the support samples as points in the feature space, estimates the class prototype by the mean value of the support features, and there is a great class bias in the small sample scene, which leads to inaccurate image detection. SUMMARY

[0005] In view of the above defects or improvement needs of the prior art, the present application provides a small sample image detection model training method and an image detection method, which aims to solve the technical problem that the traditional meta-learning model has a great class bias in the small sample scene, leading to inaccurate image detection.

[0006] To achieve the above purpose, according to one aspect of the present application, a small sample image detection model training method is provided, comprising:

[0007] S1: initialize an original image detection model, the original image detection model comprising an aggregation module and a support branch, a query branch and a detection head connected thereto; the support branch comprising, connected in sequence: a first backbone network and a variational autoencoder; the query branch comprising, connected in sequence: a second backbone network and a region generation network; the first backbone network and the second backbone network being parameter twins;

[0008] S2: train the original image detection model using a support image set, a labeled query image D q and each target class c in the class M until the total loss reaches convergence, obtaining a base class image detection model;

[0009] During the training process, the following steps are performed: inputting the support image set into the first backbone network to obtain support features S; inputting the support features S and the semantic embedding e of the target class c into the encoder of the variational autoencoder to generate the mean μ and the variance σ, and inputting the mean μ and the variance σ into the decoder of the variational autoencoder to obtain generated features S'; extracting the latent variable z = μ + σ from the variational autoencoder; inputting the labeled query image D q into the second backbone network to obtain query features, and inputting the query features into the region generation network to obtain target proposals including positive samples and negative samples; performing RoIAlign operation on the target proposals to obtain RoI features; performing aggregation on the latent variable z and the RoI features to obtain fusion features f; inputting the fusion features f into the detection head to obtain predicted classes and predicted boxes The total classification loss L is represented by the sum of the classification loss of the positive samples and the classification loss of the negative samples cls The regression loss L is calculated using the predicted boxes and the original label b reg The reconstruction loss L is represented by the difference between the support features S and the generated features S' rec The KL loss L is calculated using the mean μ and the variance σ KL The semantic consistency loss L is represented by the difference between the semantic embedding e and the support features S sem The total loss L is calculated using the formula L = L cls + L reg + L rec +m1·L KL +m2·L sem m1 and m2 are hyperparameters of L KL and L sem respectively.

[0010] In one of the embodiments, the calculation process of the total classification loss L cls is as follows:

[0011] Select n negative samples as instance negative samples;

[0012] use Calculate the RoI features f of the negative sample of the j-th instance. j With the k-th class prototype S k Cosine similarity between use Calculate the foreground probability p of the negative sample of the j-th instance. j M is the total number of class prototypes; using w j =(1-p j ) γ ·exp(-λH(p j )) Calculate the classification weight w of the j-th negative sample. j H(p) j ) is p j The information entropy, where λ is a hyperparameter and γ is a regulation factor;

[0013] use Calculate the classification loss for negative samples; use L cls =L pos +L neg Calculate the total classification loss L cls c j and Let L be the true label and the predicted label of the negative sample of the j-th instance. pos Classification loss for positive samples.

[0014] In one embodiment, selecting n samples from the negative samples as instance negative samples includes:

[0015] negative samples N bg Divided into true negative samples N t and false negative samples N f ;

[0016] From the true negative samples N respectively t and the pseudo-negative sample N f A group is sampled from the middle and combined together as the instance negative sample N. sample .

[0017] In one embodiment, the true negative samples N are respectively... t and the pseudo-negative sample N f A sample group is included:

[0018] For true negative samples N t Sampling is performed using IoU balanced sampling;

[0019] For false negative samples Nf Random sampling was used for sampling.

[0020] In one embodiment, the negative sample N bg Divided into true negative samples N t and false negative samples N f This includes: using formulas From negative samples N bg Select N true negative samples t The remaining part is used as a false negative sample N. f Where g represents the bounding box, G represents the set of bounding boxes, iou(·,·) represents the standard intersection-union ratio metric, and τ represents the maximum threshold for distinguishing between true and false negative samples.

[0021] In one embodiment, the step of aggregating the latent variable z and the RoI features to obtain the fused feature f includes: using f = F agg ([F1(q roi ⊙z k ),F2(q roi -z k ),q roi ]) The k-th latent variable z k and the i-th RoI feature q roi Perform aggregation; F1, F2, F agg For a single, independent fully connected layer, ⊙ represents a channel-level multiplication operation.

[0022] In one embodiment, the semantic consistency loss L is represented by the difference between the target category c and the supporting feature S. sem This includes: using formulas Calculate the cosine similarity between the semantic embedding e and the supporting feature S to represent the semantic consistency loss L. sem , where e T Let ||·|| be the transpose of the semantic embedding vector e, and ||·|| denote the paradigm.

[0023] According to another aspect of the present invention, an image detection method is provided, comprising:

[0024] If the category to be identified in the image set to be detected belongs to class M, then the image set to be detected is input into the trained base class image detection model to find the target detection image set and mark the label and location of the target subject.

[0025] In one embodiment, the image detection method further includes:

[0026] If the category to be identified in the image set to be detected does not belong to class M, then the few-sample support image set and the already labeled query image D are used. qand target category c' to train the base class image detection model until the total loss reaches convergence, obtaining a target image detection model; the to-be-identified category belongs to the target category c';

[0027] inputting the to-be-detected image set into the target image detection model, finding out a target detection image set from the target image detection model, and marking the label and position of the target subject.

[0028] In one of the embodiments, the training process of the target image detection model is the same as that of the base class image detection model.

[0029] According to another aspect of the present application, a base class training device of an image detection model is provided, comprising:

[0030] an initialization module configured to initialize an original image detection model, wherein the original image detection model comprises an aggregation module and a support branch, a query branch and a detection head connected thereto; the support branch comprises, in sequence, a first backbone network and a variational autoencoder; the query branch comprises, in sequence, a second backbone network and a region generation network; the first backbone network and the second backbone network are parameter-twinning;

[0031] a training model configured to train the original image detection model by using a support image set, a labeled query image D q and each target category c in category M until the total loss reaches convergence, obtaining a base class image detection model;

[0032] In the training process, the following steps are performed: inputting the support image set into the first backbone network to obtain support features S; inputting the support features S and the semantic embedding e of the target category c into an encoder of the variational autoencoder to obtain a mean value μ and a variance σ, and inputting the mean value μ and the variance σ into a decoder of the variational autoencoder to obtain generated features S'; extracting a latent variable z = μ + σ from the variational autoencoder; inputting the labeled query image D q into the second backbone network to obtain query features, and inputting the query features into the region generation network to obtain target proposals comprising positive samples and negative samples; performing a RoIAlign operation on the target proposals to obtain RoI features; aggregating the latent variable z and the RoI features to obtain fusion features f; and inputting the fusion features f into the detection head to obtain predicted categories and predicted boxes wherein the total classification loss L is represented by the sum of the classification loss of the positive samples and the classification loss of the negative samples cls , the regression loss L is calculated by using the predicted boxes and the original label b reg , and the reconstruction loss L is represented by the difference between the support features S and the generated features S'.rec , the KL loss L is calculated by using the mean μ and the variance σ KL ; the semantic consistency loss L is represented by using the difference between the semantic embedding e and the support feature S sem ; the total loss L is calculated by using the formula L = L cls + L reg + L rec + m1·L KL + m2·L sem ; m1 and m2 are hyperparameters of L KL and L sem , respectively.

[0033] According to another aspect of the present application, an image detection device is provided, comprising: if a to-be-recognized category of a to-be-detected image set belongs to a category M, inputting the to-be-detected image set into the base-class image detection model, finding a target detection image set therefrom, and marking a label and a position of a target subject.

[0034] According to another aspect of the present application, an image processing device is provided, comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the base-class training method or the image detection method when executing the computer program.

[0035] According to another aspect of the present application, a computer readable storage medium is provided, storing a computer program, and the computer program implementing the steps of the base-class training method or the image detection method when executed by a processor.

[0036] Overall, compared with the prior art, the above technical solutions conceived by the present application can achieve the following beneficial effects:

[0037] (1) The present application provides a training method of a small sample image detection model, which trains the original image detection model by using a support image set, a labeled query image D q and each target category c in the category M until the total loss reaches convergence, to obtain a base-class image detection model; inputs the support feature S and the semantic embedding e of the target category c into a variational autoencoder VAE, and compared with the support feature and the reconstructed feature of a specific category, the latent vector in the intermediate process of the VAE contains more general features of the category prototype; samples the latent vector containing semantic information from the distribution and performs feature aggregation with the query feature, so that the estimated new category center is closer to the real category center in a small sample scenario; in addition, the total loss is composed of the reconstruction loss, the KL divergence, the classification loss, the regression loss and the semantic consistency loss in the VAE, which can greatly improve the training precision.

[0038] (2) The present application uses the sum of the classification loss of the positive sample and the classification loss of the negative sample to represent the total classification loss L clsWhere n is selected from the negative samples as an instance negative sample as a representative, the RoI feature is calculated and the cosine similarity of the sample class prototype is obtained to obtain the foreground probability of the negative sample proposal, and the gradient of the negative sample proposal is weighted according to the foreground probability to reduce the weight of the false negative sample, while maintaining the contribution of the hard negative sample to the model training, so that the model is better generalized to new classes. From the perspective of gradient optimization, the influence of false negative samples is reduced, and the target is avoided to be incorrectly classified as background in directional retrieval.

[0039] (3) The negative sample N bg is divided into true negative samples N t and false negative samples N f ; a group is sampled from the true negative samples N t and the false negative samples N f , considering that both true negative samples and false negative samples have high similarity to the class prototype, and the sampled instance negative samples are more representative.

[0040] (4) For the false negative samples N f , random sampling is used. The true negative samples N t with high IoU are more likely to become difficult negative samples, and IoU balanced sampling is used for sampling to guide the distribution of difficult negative samples, which can improve the sampling accuracy.

[0041] (5) The negative samples obtained by the region proposal allocation include difficult negative samples and false negative samples, and most of the potential positive samples have an IoU lower than the threshold, such as 0.1, between the true label. Therefore, a proper IoU threshold is set to distinguish between true negative samples N t and false negative samples N f . The distinguishing process of each negative sample n can be expressed as: Where N bg represents the background proposal set obtained in the small sample incomplete annotation scene, iou(·) represents the standard IoU metric, and τ represents the set IoU threshold for distinguishing true and false negative samples.

[0042] (6) The translation-invariant class prototype center obtained by sampling and the RoI feature obtained by the query branch are aggregated in the scheme. Unlike direct multiplication fusion, element multiplication and subtraction are performed here to improve their relationship with new class features, and then the original RoI feature is spliced to reduce the noise caused by random sampling. The independent fully connected layers F1 and F2 are activated by the ReLU function, and the fully connected layer F agg Finally, the features are aggregated to obtain features carrying more image features.

[0043] (7) The scheme uses the formula A cosine similarity of the semantic embedding e and the support feature S is calculated to represent a semantic consistency loss L sem , by cross-modal consistency alignment, to overcome the heterogeneity of class name semantic concepts and class prototype visual features. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 is a flowchart of the training method of the small sample image detection model provided in Embodiment 1 of the present application;

[0045] Figure 2 is a structural schematic diagram of the original image detection model in the training method of the small sample image detection model provided in Embodiment 1 of the present application;

[0046] Figure 3 is a structural schematic diagram of the semantic-guided conditional variational autoencoder provided in Embodiment 1 of the present application;

[0047] Figure 4 is a structural schematic diagram of the negative sample gradient optimization module provided in Embodiment 1 of the present application. DETAILED DESCRIPTION

[0048] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0049] Embodiment 1

[0050] The present embodiment provides a training method of a small sample image detection model, as shown in Figure 1 S1: initializing an original image detection model, as shown in Figure 2 The original image detection model includes an aggregation module and a support branch, a query branch and a detection head connected thereto; the support branch includes a first backbone network and a variational autoencoder connected in sequence; the query branch includes a second backbone network and a region generation network connected in sequence; the first backbone network and the second backbone network are parameter twins; S2: training the original image detection model using a support image set, a labeled query image D q and each target class c in the class M until the total loss reaches convergence, obtaining a base class image detection model.

[0051] In the training process, the following steps are performed: inputting a support image set into a first backbone network for feature extraction to obtain support features S; inputting the support features S and the semantic embedding e of the target class c into an encoder of a variational autoencoder to generate a mean value μ and a variance σ, inputting the mean value μ and the variance σ into a decoder of the variational autoencoder to obtain generated features S'; extracting a latent variable z = μ + σ from the variational autoencoder; inputting the labeled query image D q into a second backbone network for feature extraction to obtain query features, inputting the query features into a region generation network to obtain target proposals including positive samples and negative samples; performing a RoIAlign operation on the target proposals to obtain RoI features; aggregating the latent variable z and the RoI features to obtain fusion features f; inputting the fusion features f into a detection head to obtain a predicted class and a predicted box The total classification loss L is represented by the sum of the classification loss of the positive sample and the classification loss of the negative sample cls The regression loss L is calculated by using the predicted box and the original label b reg The reconstruction loss L is represented by the difference between the support features S and the generated features S' rec The KL loss L is calculated by using the mean value μ and the variance σ KL The semantic consistency loss L is represented by the difference between the semantic embedding e and the support features S sem The total loss L is calculated by using the formula L = L cls + L reg + L rec + m1·L KL + m2·L sem m1 and m2 are hyperparameters of L KL and L sem , respectively.

[0052] Regarding the support branch, first, image enhancement is performed on the support image. The search space of the image enhancement strategy contains K sub-strategies, each of which is composed of N operations applied to a single image in turn. In the training process, an image enhancement strategy is randomly selected and applied to the current image. The enhanced image and the original image together form a support set, and a twin backbone network is used to extract features from the support image and the query image. The support image is subjected to average pooling to obtain class-related support features S. Then, the C-VAE is trained to combine the features of the support image with the semantic embedding of the class, and map them to the latent space. Finally, the latent vector extracted from the C-VAE contains not only visual information but also semantic information of the class, enabling the model to have good generalization ability for new classes in a small sample scenario.

[0053] The semantic-guided conditional variational autoencoder aims to convert the support feature S into a class distribution N and extract latent vector z from N for feature aggregation. The module structure is shown in Figure 3 emb The semantic embedding of the class is extracted from the base dataset. For a given class c k , its semantic embedding e k is extracted as the conditional input of the VAE:

[0054] e k =F emb (c k );

[0055] Assuming that the latent vector z is generated by the prior distribution p(z|e k ), the optimization goal is to maximize the conditional log-likelihood logp(S k |e k ), that is, for a given class semantic embedding e k and latent vector z, the support feature S k can be generated. Specifically, the approximate posterior distribution q φ (z|S k , e k ) is introduced to approximate the true posterior distribution p(z|e k ) by minimizing the KL divergence, which is equivalent to optimizing the variational lower bound (ELBO):

[0056] Let the posterior distribution be a multivariate Gaussian distribution q φ (z|S k , e k ) = N(μ, σ), and the parameters μ and σ are obtained by the encoder F enc . Through the reparameterization trick, the latent vector z = μ + σ·ε is obtained, where ε ~ N(0, 1), and then the decoder is used to generate the reconstruction target S'. The first term on the right side can be simplified as the reconstruction loss:

[0057] L rec =||S-S'||=||S-F dec (z)||.

[0058] To ensure that the generated features and the semantic embedding of the class are consistent in the semantic space, thereby better reflecting the class information. In addition to the reconstruction loss and the KL loss, the semantic consistency loss L k between S k and e sem is calculated using the cosine similarity, thereby improving the model's understanding of the sample class.

[0059] ​The query branch extracts features Q from the query image using the same Siamese backbone network as the support branch. It then utilizes RPN and RoI operations to obtain RoI features, where each RoI feature represents visual information about a potential target object region. The query branch aggregates these RoI features with the latent vectors sampled by the support branch. This aggregation operation employs a non-linear fusion network, and the result serves as the input to the detection head.

[0060] Furthermore, the total classification loss L cls The calculation process is as follows: Select n negative samples from the negative samples as instance negative samples; use Calculate the RoI features f of the negative sample of the j-th instance. j With the k-th class prototype S k Cosine similarity between use Calculate the foreground probability p of the negative sample of the j-th instance. j M is the total number of class prototypes; using w j =(1-p j ) γ ·exp(-λH(p j )) Calculate the classification weight w of the j-th negative sample. j H(p) j ) is p j Information entropy, where λ is a hyperparameter; utilizing Calculate the classification loss for negative samples; use L cls =L pos +L neg Calculate the total classification loss L cls c j and Let L be the true label and the predicted label of the negative sample of the j-th instance. pos Classification loss for positive samples.

[0061] In this process, true and false negative samples have been separated during negative sample discrimination and sampling. For false negative samples, to reduce their impact and avoid overfitting the model to the training samples under small sample conditions, the gradient is optimized. This is done for an M-way K-shot task and the class-prototype support features s obtained in the support branch. j The gradient of a proposal is adjusted by calculating the prospect probability of the proposal.

[0062] like Figure 4 As shown, for the background proposal n in the false negative sample proposal, RoI features are generated through the RoIAlign operation, and then the pooled RoI features f are calculated. i With class prototype p t The cosine similarity between them is: For a total of M classes, the highest similarity score is selected as the result, and the foreground probability of the i-th negative sample sampling proposal can be expressed as: According to the foreground probability p i , the weight w of the re-calculated negative sample can be calculated, so that the gradient of the background sample with low similarity to the class prototype will not be significantly reduced, and the false negative sample with high foreground probability is given a lower weight, and the weight calculation can be expressed as:

[0063] w = (1-p) γ ·exp(-λH(p));

[0064] H(p) = -plog(p)-(1-p)log(1-p);

[0065] where γ is a tuning factor to control the decay rate of the false negative sample weight. H(p) is the information entropy, which is used to capture the uncertainty of the probability. When p tends to 0 or 1, the entropy value is low, and the exponential decay term has less effect on the weight. When p is close to the intermediate value 0.5, the entropy value is higher, that is, the sample as the foreground is more uncertain, and the exponential decay term will effectively reduce the weight fluctuation. λ is a hyperparameter, which is the weight coefficient of entropy.

[0066] Further, n are selected from the negative samples as instance negative samples, including: dividing the negative samples N bg into true negative samples N t and false negative samples N f ; a group is sampled from the true negative samples N t and the false negative samples N f respectively, and combined together as the instance negative samples N sample . Further, a group is sampled from the true negative samples N t and the false negative samples N f , including: for the true negative samples N t , IoU balanced sampling is used for sampling; for the false negative samples N f , random sampling is used for sampling.

[0067] The true negative samples and the false negative samples are sampled respectively. For the false negative samples N f , random sampling is used for extraction. The true negative samples N t with high IoU are more likely to become difficult negative samples, and IoU balanced sampling is used for sampling to guide the distribution of difficult negative samples. Specifically, considering sampling N t proposals from N sample , according to the IoU range between the sample and the true box, it is divided into K intervals. Then the probability p k that the sample is selected in the k-th interval is: where Ck denotes the number of sampling candidates within the corresponding interval k. The required N sample negative samples are uniformly distributed to each interval, and then samples are uniformly selected from these distributed intervals.

[0068] Further, the negative samples N bg are divided into true negative samples N t and false negative samples N f , including: selecting the true negative samples N bg from the negative samples N t using the formula , and the remaining part as the false negative samples N f ; wherein g represents the bounding box, G represents the bounding box set, iou(·,·) represents the standard intersection over union metric, and τ represents the maximum threshold for distinguishing true and false negative samples.

[0069] Among the negative samples obtained by the region proposal distribution, both the difficult negative samples and the false negative samples have high similarity to the class prototypes, and most of the potential positive samples have an IoU lower than 0.1 with the true label, therefore, a proper IoU threshold is set to distinguish the true negative samples N t and the false negative samples N f . The distinguishing process of each negative sample n can be expressed as:

[0070]

[0071] N bg represents the background proposal set obtained under the small sample incomplete annotation scene, iou(·) represents the standard IoU metric, and τ represents the set for distinguishing true and false negative samples.

[0072] Further, the latent variable z and the RoI feature are aggregated to obtain the fusion feature f, including: aggregating the kth latent variable z agg and the ith RoI feature q roi using f=F k ([F1(q roi ⊙z k ),F2(q roi -z k ),q roi ]); F1, F2, F agg are single independent fully connected layers, and ⊙ represents a channel-level multiplication operation.

[0073] Further, the semantic consistency loss L sem is represented by the difference between the target class c and the support feature S, including: calculating the cosine similarity between the semantic embedding e and the support feature S using the formula to represent the semantic consistency loss L sem , wherein eT Let ||·|| be the transpose of the semantic embedding vector e, and ||·|| denote the paradigm.

[0074] Feature aggregation is performed on supporting features and query features. The approximate class distribution N(μ) is found in the supporting branches. k ,σ k ), from which the potential vector z is sampled k As class centers, a nonlinear fusion network is used to aggregate the translation-invariant class prototype centers obtained from sampling and the RoI features obtained from the query branch. Unlike direct multiplication fusion, element-wise multiplication and subtraction are performed first to improve their relationship with the new class features, and then they are concatenated with the original RoI features to reduce noise from random sampling. Independent fully connected layers F1 and F2 are activated by the ReLU function, and then the fully connected layer F1 is used to further enhance the features. agg Finally, feature aggregation is performed for class prototype z. k and RoI features q roi The entire process can be represented as follows:

[0075] f = F agg ([F1(q roi ⊙z k ),F2(q roi -z k ),q roi ]).

[0076] The classification loss, optimized using the weights w obtained from the negative sample gradient optimization module, can be expressed as:

[0077]

[0078] Where L pos and L neg Let represent the classification loss for positive and negative samples, respectively. Let m and n be the number of positive and negative samples, respectively. Cross-entropy is used to compare the predicted class and the classification target c. i and The calculation is performed. The optimization objective is to find the parameters θ of the optimal classifier. The gradient descent process can be represented as:

[0079]

[0080] Where γ is the learning rate, and when the negative sample is a true negative sample, the sample weight w is... j The weights are set to 1, but when negative samples are false, the sample weights can be adjusted. This aims to maintain the contribution of hard negative samples to model training while reducing the weights of false negative samples.

[0081] Example 2

[0082] The embodiment provides an image detection method, comprising the following steps: if a to-be-recognized category of a to-be-detected image set belongs to a category M, inputting the to-be-detected image set into a base-class image detection model trained in advance, finding out a target detection image set from the to-be-detected image set, and marking a label and a position of a target subject.

[0083] Further, the image detection method further comprises the following steps: if the to-be-recognized category of the to-be-detected image set does not belong to the category M, training the base-class image detection model by using a few-shot support image set, a labeled query image D q and a target category c until a total loss reaches convergence, obtaining a target image detection model, the to-be-recognized category belonging to the target category c, inputting the to-be-detected image set into the target image detection model, finding out the target detection image set from the to-be-detected image set, and marking the label and the position of the target subject.

[0084] Further, a training process of the target image detection model is the same as that of the base-class image detection model. The base-class image detection model has the same structure as that of an original image detection model, and comprises an aggregation module and a support branch, a query branch and a detection head connected with the aggregation module; the support branch comprises, connected in sequence, a first backbone network and a variational autoencoder; the query branch comprises, connected in sequence, a second backbone network and a region generation network; the first backbone network and the second backbone network are parameter-twinning; the training process of the target image detection model described below is all based on the structure of the base-class image detection model.

[0085] In the training process, the following steps are performed: inputting the support image set into the first backbone network to perform feature extraction to obtain support features S; inputting the support features S and the semantic embedding e of the target category c into an encoder of the variational autoencoder to generate a mean value μ and a variance σ, inputting the mean value μ and the variance σ into a decoder of the variational autoencoder to obtain generated features S'; extracting a latent variable z = μ + σ from the variational autoencoder; inputting the labeled query image D q into the second backbone network to perform feature extraction to obtain query features, inputting the query features into the region generation network to obtain target proposals comprising positive samples and negative samples; performing a RoIAlign operation on the target proposals to obtain RoI features; performing aggregation on the latent variable z and the RoI features to obtain fusion features f; inputting the fusion features f into the detection head to obtain a predicted category and a predicted bounding box The total classification loss L cls is represented by a sum of a classification loss of the positive samples and a classification loss of the negative samples. reg The regression loss L is calculated by using the predicted bounding box and the original label b. rec The reconstruction loss L KL; the semantic consistency loss L is represented by the difference between the semantic embedding e and the support feature S sem ; the formula L = L cls + L reg + L rec + m1·L KL + m2·L sem is used to calculate the total loss L; m1 and m2 are hyperparameters of L KL and L sem respectively.

[0086] Embodiment 3

[0087] The embodiment provides a base class training device of an image detection model, comprising: an initialization model, configured to initialize an original image detection model, the original image detection model comprising an aggregation module and a support branch, a query branch and a detection head connected thereto; the support branch comprises, connected in sequence: a first backbone network and a variational autoencoder; the query branch comprises, connected in sequence: a second backbone network and a region generation network; the first backbone network and the second backbone network are parameter twins; a training model is configured to train the original image detection model by using a support image set, a labeled query image D q and each target class c in the class M until the total loss reaches convergence, to obtain a base class image detection model;

[0088] In the training process, the following steps are performed: inputting the support image set into the first backbone network to extract support features S; inputting the support features S and the semantic embedding e of the target class c into the encoder of the variational autoencoder to generate the mean value μ and the variance σ, and inputting the mean value μ and the variance σ into the decoder of the variational autoencoder to obtain generated features S'; extracting the latent variable z = μ + σ from the variational autoencoder; inputting the labeled query image D q into the second backbone network to extract query features, and inputting the query features into the region generation network to obtain target proposals including positive samples and negative samples; performing RoIAlign operation on the target proposals to obtain RoI features; aggregating the latent variable z and the RoI features to obtain fusion features f; inputting the fusion features f into the detection head to obtain predicted classes and predicted boxes The total classification loss L cls is represented by the sum of the classification loss of the positive samples and the classification loss of the negative samples, the regression loss L reg is calculated by using the predicted boxes and the original label b, the reconstruction loss L rec is represented by the difference between the support features S and the generated features S', the KL loss L KL is calculated by using the mean value μ and the variance σ, and the semantic consistency loss L sem is represented by the difference between the semantic embedding e and the support feature S; and the formula L = Lcls +L reg +L rec +m1·L KL +m2·L sem The total loss L is calculated; m1 and m2 are hyperparameters of L KL and L sem .

[0089] Where the few-shot target detection needs to use a small number of k shots to predict new classes, in the case of data scarcity and large sample changes, if only the support samples are encoded as a single feature vector, it is difficult to represent the entire class distribution. First, the support image can be enhanced, specifically, the enhancement operation includes color operation, geometric operation, etc. The selection of the enhancement operation is defined as a search problem, and the search space contains K sub-strategies, each sub-strategy consists of N operations applied to a single image in order, and in the training process, the image enhancement strategy is randomly selected and applied to the current image. The enhanced image and the original image are used as support images to extract features via the backbone network as the input of the variational autoencoder (VAE). VAE learns the probability distribution of the data by embedding the data into the latent space and reconstructing the input data from the space.

[0090] This embodiment adopts conditional variational prototype modeling (C-VAE), first, the pre-trained word embedding model is used to extract the semantic embedding of the class from the base class data set as the conditional input, C-VAE combines the features of the support image with the semantic embedding of the class and maps it to the latent space. Through the C-VAE decoder, the reconstructed features are generated. In the inference stage, the model estimates the distribution of the new class using the distribution learned by VAE on the base class data. Compared with the support features and reconstructed features of a specific class, the latent vector in the intermediate process of VAE contains more general features of the class prototype, therefore, the latent vector containing semantic information is sampled from the distribution, and it is aggregated with the query feature, so that the estimated new class center is closer to the true class center.

[0091] On the other hand, in the training sample images of the FSOD, only the instances of the current target category are labeled, while in fact there may be multiple unlabeled instances in the image, which are regarded as negative samples (background). These false negative samples can mislead the classifier and cannot correctly distinguish new class targets from background. In order to reduce the influence of false negative samples, an optimization method for negative samples is introduced in the query branch. First, the query sample is input into the region candidate network (RPN) to generate region proposals, which are assigned as positive samples (foreground) and negative samples (background). Among these negative samples, there are difficult negative samples and false negative samples, both of which have high similarity to the class prototype. By setting an IoU threshold, the two can be distinguished. Given a negative sample proposal, RoI features are generated by RoI-aligned pooling, the cosine similarity between the RoI features and the class prototype obtained from the support sample is calculated to obtain the foreground probability of the proposal, and the gradient of the negative sample proposal is weighted according to the foreground probability to reduce the weight of the false negative sample, while maintaining the contribution of the hard negative sample to the model training, so that the model can better generalize to new classes. Finally, the reconstruction loss in VAE, the KL divergence, the classification loss in the detection head, the regression loss and the consistency loss constitute the total loss.

[0092] Embodiment 4

[0093] The embodiment provides an image detection device, which comprises: if a to-be-recognized category of a to-be-detected image set belongs to a class M, inputting the to-be-detected image set into a base class image detection model, finding out a target detection image set from the base class image detection model, and marking a label and a position of a target subject.

[0094] Embodiment 5

[0095] The embodiment provides an image processing device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the steps of the base class training method or the image detection method when executing the computer program.

[0096] Embodiment 6

[0097] The embodiment provides a computer readable storage medium, which stores a computer program, and the computer program implements the steps of the base class training method or the image detection method when executed by a processor.

[0098] Those skilled in the art will easily understand that the above description is only the preferred embodiment of the present application, and is not intended to limit the present application, and any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for training a small sample image detection model, characterized in that, Comprising: S1: initialize an original image detection model, the original image detection model comprising an aggregation module and a support branch, a query branch and a detection head connected thereto; The support branch comprises a first backbone network and a variational autoencoder connected in turn; the query branch comprises a second backbone network and a region generation network connected in turn; the first backbone network and the second backbone network are parameter twins; S2: utilize the support image set, the labeled query image and each target class c in the class M, train the original image detection model until the total loss reaches convergence, obtaining a base class image detection model; The training process involves the following steps: inputting the support image set into the first backbone network for feature extraction to obtain support features S; inputting the support features S and the semantic embedding e of the target category c into the encoder of the variational autoencoder to generate the mean. and variance , the mean and variance The decoder of the variational autoencoder is input to obtain generated features S'; latent variables are extracted from the variational autoencoder. ; tagged query images The query features are obtained by inputting the second backbone network for feature extraction. These query features are then input into the region generation network to obtain target proposals including positive and negative samples. The RoIAlign operation is performed on the target proposals to obtain RoI features. The latent variable z and the RoI features are aggregated to obtain a fused feature f. The fused feature f is then input into the detection head to obtain the predicted category. and prediction boxes Using prediction boxes Calculate the regression loss with the original label b. Utilizing the aforementioned support features The difference between the generated feature S' and the original feature S' represents the reconstruction loss. Using the mean and variance Calculate KL loss ; utilizing the semantic embedding e and the supporting features The difference represents the semantic consistency loss. Using formulas Calculate total loss ; and They are respectively and hyperparameters; Total classification loss The calculation process is as follows: n is selected from the negative samples as an instance negative sample; the RoI feature of the jth instance negative sample is calculated by using The cosine similarity between the jth instance negative sample and the kth class prototype ; the foreground probability of the jth instance negative sample is calculated by using M is the total number of class prototypes; the classification weight of the jth instance negative sample is calculated by using ; The information entropy of λ is a hyperparameter, is an adjustment factor; the classification loss of the negative sample is calculated by using ; the total classification loss is calculated by using , and are the real label and the predicted label of the jth instance negative sample, the classification loss of the positive sample, is the cross entropy;​​​​​ The step of selecting n as instance negative samples from the negative samples includes: selecting n negative samples as instance negative samples. Divided into true negative samples and false negative samples ; respectively from the true negative samples and the pseudo-negative samples A group is sampled from the middle and combined together as an instance negative sample. ; The method comprises the following steps of: sampling a group from the true negative samples and the false negative samples respectively, comprising: for the true negative samples , using IoU balanced sampling to sample; for the false negative samples , using random sampling to sample; The negative sample is divided into true negative samples and false negative samples , comprising: selecting true negative samples from negative samples by using a formula , and the remaining part as false negative samples ; wherein, denotes a bounding box, denotes a bounding box set, denotes a standard intersection over union metric, denotes a maximum threshold value for distinguishing true and false negative samples. 2.The method of claim 1, wherein, The aggregation of the latent variable z and the RoI feature to obtain a fusion feature f comprises: Utilizing the kth latent variable and the ith RoI feature are aggregated; , , for a single independent fully connected layer, denotes a channel-wise multiplication operation. 3.The method of Claim 1, wherein The difference between the target category and the support feature represents semantic consistency loss , comprising: The semantic consistency loss is represented by the cosine similarity of the semantic embedding e and the support feature computed using the formula where is the transpose of the semantic embedding vector denotes the norm.​​ 4. An image detection method characterized by, Comprising: If the to-be-detected image set belongs to class M, input the to-be-detected image set into the base class image detection model trained according to any one of claims 1-3, find out a target detection image set therefrom, and mark the label and position of the target subject.

5. The image detecting method of claim 4, wherein, Further comprising: If the category to be identified in the image set to be detected does not belong to class M, then the few-sample support image set and the already labeled query image are used. The base class image detection model is trained with the target class c' until the total loss converges, thus obtaining the target image detection model; the category to be identified belongs to the target class c'. Input the to-be-detected image set into the target image detection model, find out a target detection image set therefrom, and mark the label and position of the target subject.

6. The image detecting method of claim 5, wherein, The training process of the target image detection model is the same as that of the base class image detection model.

Images