Training method of small sample image detection model and image detection method

Abstract

The invention discloses a training method of a small sample image detection model and an image detection method, and belongs to the technical field of deep learning, and the base class training method comprises the steps: training an original image detection model through employing a support image set, a marked query image, and each target class in a class until the total loss is converged, obtaining a base class image detection model; support features and semantics of a target category are embedded and input into a variational auto-encoder, and compared with support features and reconstruction features of a specific category, potential vectors in a VAE intermediate process contain general features of more types of prototypes; potential vectors containing semantic information are sampled from the distribution, and feature aggregation is carried out on the potential vectors and query features, so that the estimated new class center is closer to a real class center; in addition, the reconstruction loss, the KL divergence, the classification loss, the regression loss and the semantic consistency loss in the VAE form the total loss, so that the training precision can be greatly improved.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology, and more specifically, relates to a training method for a small sample image detection model and an image detection method. Background Art

[0002] Directed web image retrieval is mainly used to screen and extract images containing specific target objects from massive images. However, in the face of massive web images, usually only a small number of images can be annotated. This small sample phenomenon is more typical for new things or non-hot events on the Internet. In recent years, object detection technology based on deep learning has been widely studied. Object detection aims to identify and accurately locate objects in images, and usually requires a large amount of annotated data for training.

[0003] In real scenarios, it is difficult and costly to obtain large-scale labeled samples. Therefore, the few-shot object detection (FSOD) method in data-scarce conditions has also received increasing attention. The main idea of ​​FSOD is to train a base class detection model on a base class dataset with a large amount of labeled information, and then use a new class dataset with very few annotations and the prior knowledge provided by the base class model to detect new classes. Through FSOD, the model can effectively filter out images containing target objects from a large number of unlabeled images when there are only a few labeled images. At present, the meta-learning method in the paradigm of few-shot object detection trains the model by simulating multiple few-shot tasks, so that it can quickly adapt to the detection task of new classes. Meta-learning obtains task-level knowledge pre-trained on the base class through scenario training, enhancing the generalization ability of the model when encountering new classes.

[0004] However, since there are only K samples for each class, the meta-learning method is highly dependent on the quality of the support set. Traditional meta-learning models regard support samples as points in the feature space and use the mean of the support features to estimate the class prototype. In small sample scenarios, there is a large class bias, resulting in inaccurate image detection. Summary of the invention

[0005] In response to the above defects or improvement needs of the prior art, the present invention provides a training method for a small sample image detection model and an image detection method, which aims to solve the technical problem that traditional meta-learning models have large class biases in small sample scenarios, resulting in inaccurate image detection.

[0006] To achieve the above object, according to one aspect of the present invention, a training method for a small sample image detection model is provided, comprising:

[0007] S1: Initialize the original image detection model, which includes an aggregation module and a support branch, a query branch and a detection head connected thereto; the support branch includes: a first backbone network and a variational autoencoder connected in sequence; the query branch includes: a second backbone network and a region generation network connected in sequence; the first backbone network and the second backbone network parameter twins;

[0008] S2: Using the support image set and the labeled query image D q and each target category c in class M to train the original image detection model until the total loss reaches convergence, thereby obtaining a base class image detection model;

[0009] The following steps are performed during the training process: the support image set is input into the first backbone network for feature extraction to obtain support features S; the support features S and the semantic embedding e of the target category c are input into the encoder of the variational autoencoder to generate mean μ and variance σ, and the mean μ and variance σ are input into the decoder of the variational autoencoder to obtain generated features S'; the latent variable z = μ + σ is extracted from the variational autoencoder; the labeled query image D q Input the second backbone network to extract features to obtain query features, input the query features to the region generation network to obtain target proposals including positive samples and negative samples; perform RoIAlign operation on the target proposal to obtain RoI features; aggregate the latent variable z and the RoI features to obtain fused features f; input the fused features f to the detection head to obtain a predicted category and prediction box The total classification loss L is expressed as the sum of the classification loss of positive samples and the classification loss of negative samples. cls , using the prediction box Calculate the regression loss L with the original label b reg , using the difference between the support feature S and the generated feature S' to represent the reconstruction loss L rec , calculate the KL loss L using the mean μ and variance σ KL ; Using the difference between the semantic embedding e and the supporting feature S to represent the semantic consistency loss L sem ; Using the formula L = L cls +L reg +L rec +m1·L KL +m2·L sem Calculate the total loss L; m1 and m2 are L KL and L sem Hyperparameters of .

[0010] In one embodiment, the total classification loss L cls The calculation process is as follows:

[0011] Select n negative samples from the negative samples as instance negative samples;

[0012] use Calculate the RoI feature f of the jth instance negative sample j With the k-th class prototype S k The cosine similarity between use Calculate the foreground probability p of the jth instance negative sample j , M is the total number of class prototypes; using w j =(1-p j ) γ ·exp(-λH(p j )) Calculate the classification weight w of the jth instance negative sample j ; H(p j ) is p j The information entropy of is, λ is the hyperparameter, and γ is the adjustment factor;

[0013] use Calculate the classification loss of negative samples; use L cls =L pos +L neg Calculate the total classification loss L cls , c j and are the true label and predicted label of the jth instance negative sample, L pos Classification loss for positive samples.

[0014] In one embodiment, selecting n negative samples from the negative samples as example negative samples includes:

[0015] The negative sample N bg Divided into true negative samples N t and false negative samples N f ;

[0016] From the true negative samples N t And the false negative sample N f Sample a group from the sample and combine them together as the instance negative sample N sample .

[0017] In one embodiment, the true negative samples N are respectively t And the false negative sample N f A set of samples is included:

[0018] For the true negative sample N t , IoU balanced sampling is used for sampling;

[0019] For the false negative sample Nf , random sampling was used for sampling.

[0020] In one embodiment, the negative sample N bg Divided into true negative samples N t and false negative samples N f , including: using the formula From the negative sample N bg Select the true negative sample N t , the rest are taken as false negative samples N f ; where g represents the annotation box, G represents the set of annotation boxes, iou(·,·) represents the standard intersection-over-union metric, and τ represents the maximum threshold for distinguishing true and false negative samples.

[0021] In one embodiment, the step of aggregating the latent variable z and the RoI feature to obtain a fusion feature f includes: using f=F agg ([F1(q roi ⊙z k ),F2(q roi -z k ),q roi ]) The kth latent variable z k and the i-th RoI feature q roi Polymerization; F1, F2, F agg is a single independent fully connected layer, and ⊙ represents channel-level multiplication operation.

[0022] In one embodiment, the difference between the target category c and the supporting feature S is used to represent the semantic consistency loss L sem , including: using the formula Calculate the cosine similarity between the semantic embedding e and the support feature S to represent the semantic consistency loss L sem , where e T is the transpose of the semantic embedding vector e, and ||·|| represents the paradigm.

[0023] According to another aspect of the present invention, there is provided an image detection method, comprising:

[0024] If the category to be identified of the image set to be detected belongs to class M, the image set to be detected is input into the trained base class image detection model to find the target detection image set and mark the label and position of the target subject.

[0025] In one embodiment, the image detection method further includes:

[0026] If the category to be identified in the image set to be detected does not belong to class M, then use the few-sample support image set and the labeled query image D q' and target category c' to train the base image detection model until the total loss reaches convergence, thereby obtaining a target image detection model; the category to be identified belongs to target category c';

[0027] The image set to be detected is input into the target image detection model, a target detection image set is found therefrom, and the label and position of the target subject are marked.

[0028] In one of the embodiments, the training process of the target image detection model is the same as the training process of the base image detection model.

[0029] According to another aspect of the present invention, there is provided a base class training device for an image detection model, comprising:

[0030] An initialization module is used to initialize the original image detection model, wherein the original image detection model includes an aggregation module and a support branch, a query branch and a detection head connected thereto; the support branch includes: a first backbone network and a variational autoencoder connected in sequence; the query branch includes: a second backbone network and a region generation network connected in sequence; the first backbone network and the second backbone network parameter twins;

[0031] Train a model to use the support image set and the labeled query image D q and each target category c in class M to train the original image detection model until the total loss reaches convergence, thereby obtaining a base class image detection model;

[0032] The following steps are performed during the training process: the support image set is input into the first backbone network for feature extraction to obtain support features S; the support features S and the semantic embedding e of the target category c are input into the encoder of the variational autoencoder to generate mean μ and variance σ, and the mean μ and variance σ are input into the decoder of the variational autoencoder to obtain generated features S'; the latent variable z = μ + σ is extracted from the variational autoencoder; the labeled query image D q Input the second backbone network to extract features to obtain query features, input the query features to the region generation network to obtain target proposals including positive samples and negative samples; perform RoIAlign operation on the target proposal to obtain RoI features; aggregate the latent variable z and the RoI features to obtain fused features f; input the fused features f to the detection head to obtain a predicted category and prediction box The total classification loss L is expressed as the sum of the classification loss of positive samples and the classification loss of negative samples. cls , using the prediction box Calculate the regression loss L with the original label b reg , using the difference between the support feature S and the generated feature S' to represent the reconstruction loss Lrec , calculate the KL loss L using the mean μ and variance σ KL ; Using the difference between the semantic embedding e and the supporting feature S to represent the semantic consistency loss L sem ; Using the formula L = L cls +L reg +L rec +m1·L KL +m2·L sem Calculate the total loss L; m1 and m2 are L KL and L sem Hyperparameters of .

[0033] According to another aspect of the present invention, an image detection device is provided, comprising: if the to-be-detected category of the to-be-detected image set belongs to class M, the to-be-detected image set is input into the base class image detection model, a target detection image set is found therefrom, and the label and position of the target subject are marked.

[0034] According to another aspect of the present invention, there is provided an image processing device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the base class training method or the image detection method when executing the computer program.

[0035] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the base class training method or the image detection method are implemented.

[0036] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0037] (1) The present invention provides a training method for a small sample image detection model, which uses a support image set and a labeled query image D q and each target category c in class M train the original image detection model until the total loss converges to obtain a base class image detection model; the semantic embedding e of the support feature S and the target category c is input into the variational autoencoder VAE, and the latent vector of the VAE intermediate process contains more general features of the class prototype than the support features and reconstruction features of a specific class; latent vectors containing semantic information are sampled from the distribution and are feature aggregated with the query features, so that the estimated new class center is closer to the real class center in a small sample scenario; in addition, the total loss is composed of the reconstruction loss, KL divergence, classification loss, regression loss and semantic consistency loss in the VAE, which can greatly improve the training accuracy.

[0038] (2) This scheme uses the sum of the classification loss of positive samples and the classification loss of negative samples to represent the total classification loss L cls, where n negative samples are selected from the negative samples as representative examples, the RoI features and the cosine similarity of the sample class prototype are calculated to obtain the foreground probability of the negative sample proposal, and the gradient of the negative sample proposal is weighted according to the foreground probability to reduce the weight of the false negative sample, while maintaining the contribution of the hard negative sample to the model training, so that the model can be better generalized to new classes. From the perspective of gradient optimization, the impact of false negative samples is reduced to avoid misclassifying the target as background during directed retrieval.

[0039] (3) This scheme takes the negative sample N bg Divided into true negative samples N t and false negative samples N f ; From the true negative samples N t And the false negative sample N f A group of samples are sampled in the middle, considering that both true negative samples and false negative samples have class prototypes with high similarity, and sampling them separately makes the sampled instance negative samples more representative.

[0040] (4) This scheme is for false negative samples N f , extracted by random sampling. The true negative samples N with higher IoU t , is more likely to become a difficult negative sample. Using IoU balanced sampling to guide the distribution of difficult negative samples can improve sampling accuracy.

[0041] (5) This scheme considers that among the negative samples obtained by region proposal assignment, both hard negative samples and false negative samples have class prototypes with high similarity, while the IoU between most potential positive samples and the true label is lower than the threshold, such as 0.1; therefore, an appropriate IoU threshold is set to distinguish the true negative sample N t and false negative samples N f The differentiation process of each negative sample n can be expressed as: Where N bg represents the background proposal set obtained in the small sample incomplete annotation scenario, iou(·) represents the standard IoU metric, and τ represents the setting for distinguishing true and false negative samples.

[0042] (6) This scheme aggregates the sampled translation-invariant class prototype center and the RoI features obtained by the query branch. Different from direct multiplication fusion, this scheme first undergoes element-wise multiplication and subtraction to improve their relationship with the new class features, and then concatenates them with the original RoI features to reduce the noise caused by random sampling. The independent fully connected layers F1 and F2 are activated with the ReLU function and then pass through the fully connected layer F agg Finally, feature aggregation is performed, and the obtained features carry richer image features.

[0043] (7) This scheme uses the formula Calculate the cosine similarity between the semantic embedding e and the support feature S to represent the semantic consistency loss L sem , through cross-modal consistency alignment, to overcome the heterogeneity of class name semantic concepts and class prototype visual features. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of a training method for a small sample image detection model provided in Example 1 of the present invention;

[0045] Figure 2 It is a schematic diagram of the structure of the original image detection model in the training method of the small sample image detection model provided in Example 1 of the present invention;

[0046] Figure 3 is a schematic diagram of the structure of the semantically guided conditional variational autoencoder provided in Example 1 of the present invention;

[0047] Figure 4 Schematic diagram of the structure of the negative sample gradient optimization module provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0049] Example 1

[0050] This embodiment provides a training method for a small sample image detection model. Figure 1 As shown, it includes: S1: Initialize the original image detection model, such as Figure 2 As shown in FIG. 1 , the original image detection model includes an aggregation module and a support branch, a query branch and a detection head connected thereto; the support branch includes: a first backbone network and a variational autoencoder connected in sequence; the query branch includes: a second backbone network and a region generation network connected in sequence; the first backbone network and the second backbone network parameter twins; S2: using the support image set, the labeled query image D q The original image detection model is trained with each target category c in class M until the total loss converges to obtain the base class image detection model.

[0051] The following steps are performed during the training process: input the support image set into the first backbone network for feature extraction to obtain the support feature S; input the support feature S and the semantic embedding e of the target category c into the encoder of the variational autoencoder to generate the mean μ and variance σ, and input the mean μ and variance σ into the decoder of the variational autoencoder to obtain the generated feature S'; extract the latent variable z = μ + σ from the variational autoencoder; input the labeled query image D q Input the second backbone network for feature extraction to obtain query features, input the query features into the region generation network to obtain target proposals including positive samples and negative samples; perform RoIAlign operation on the target proposal to obtain RoI features; aggregate the latent variable z and RoI features to obtain fused features f; input the fused features f into the detection head to obtain the predicted category and prediction box The total classification loss L is expressed as the sum of the classification loss of positive samples and the classification loss of negative samples. cls , using the prediction box Calculate the regression loss L with the original label b reg , using the difference between the support feature S and the generated feature S' to represent the reconstruction loss L rec , calculate the KL loss L using the mean μ and variance σ KL ; The semantic consistency loss L is expressed by using the difference between the semantic embedding e and the supporting feature S sem ; Using the formula L = L cls +L reg +L rec +m1·L KL +m2·L sem Calculate the total loss L; m1 and m2 are L KL and L sem Hyperparameters of .

[0052] Regarding the support branch, first, image enhancement operations are performed on the support image. The search space of the image enhancement strategy contains K sub-strategies, each of which consists of N operations applied to a single image in sequence. During the training process, a randomly selected image enhancement strategy is applied to the current image. The enhanced image and the original image are combined into a support set. The twin backbone network is used to extract features from the support image and the query image at the same time. The support image is average-pooled to obtain the class-related support features S. Then C-VAE is trained to combine the features of the support image with the semantic embedding of the class and map them to the latent space. Finally, the latent vector extracted from C-VAE not only contains visual information, but also embeds the semantic information of the class, so that the model has good generalization ability for new classes in small sample scenarios.

[0053] Regarding the semantically guided conditional variational autoencoder, its goal is to use the semantically guided conditional variational autoencoder to transform the support feature S into the class distribution N, and extract the latent vector z from N for feature aggregation. The module structure is as follows Figure 3 As shown. First, use the pre-trained word embedding model F emb Extract the semantic embedding of the class from the base class dataset. For a given class c k , extract its semantic embedding k As conditional input to VAE:

[0054] e k =F emb (c k );

[0055] Assume that the potential vector z is given by the prior distribution p(z|e k ) is generated, and the optimization goal is to maximize the conditional log-likelihood logp(S k ∣e k ), that is, the semantic embedding e for a given class k and latent vector z, which can generate support features S k Specifically, we introduce the approximate posterior distribution q φ (z∣S k ,e k ), using the minimized KL divergence to approximate the true posterior distribution p(z|e k ), this process is equivalent to optimizing the variational lower bound (ELBO):

[0056] Let the posterior distribution be a multivariate Gaussian distribution q φ (z|S k ,e k )=N(μ,σ), the parameters μ and σ are determined by the encoder F enc Through the reparameterization technique, we get the latent vector z = μ + σ·ε, where ε ~ N(0,1), and then use the decoder to generate the reconstruction target S'. The first term on the right side can be simplified to the reconstruction loss:

[0057] L rec =||S-S'||=||SF dec (z)||.

[0058] To ensure that the semantic embedding of generated features and classes remains consistent in the semantic space, so as to better reflect the category information. In addition to the reconstruction loss and KL loss, the cosine similarity is used to calculate the metric S k and e k The semantic consistency loss L sem , thereby improving the model's ability to understand sample categories.

[0059] The query branch extracts the query image feature Q through the same twin backbone network as the support branch, and uses RPN and RoI operations to obtain the RoI features of the image. Each RoI feature represents the visual information of the potential target object area. The query branch aggregates these RoI features with the latent vector sampled by the support branch. The aggregation operation uses a nonlinear fusion network, and the result is used as the input of the detection head.

[0060] Furthermore, the total classification loss L cls The calculation process of is as follows: select n from the negative samples as instance negative samples; use Calculate the RoI feature f of the jth instance negative sample j With the k-th class prototype S k The cosine similarity between use Calculate the foreground probability p of the jth instance negative sample j , M is the total number of class prototypes; using w j =(1-p j ) γ ·exp(-λH(p j )) Calculate the classification weight w of the jth instance negative sample j ; H(p j ) is p j The information entropy of Calculate the classification loss of negative samples; use L cls =L pos +L neg Calculate the total classification loss L cls , c j and are the true label and predicted label of the jth instance negative sample, L pos Classification loss for positive samples.

[0061] Among them, true and false negative samples have been separated in negative sample discrimination and sampling. For false negative samples, in order to reduce their impact and avoid overfitting of the model to the training samples under small sample conditions, the gradient is optimized. For an M-way K-shot task and the prototype support feature s obtained in the support branch j , the foreground probability of the proposal is calculated to adjust the gradient of the proposal.

[0062] like Figure 4 As shown, for the background proposal n in the false negative sample proposal, the RoI feature is generated by the RoIAlign operation, and then the pooled RoI feature f is calculated i With the class prototype p t The cosine similarity between is: For a total of M classes, the one with the highest similarity score is selected as the result. The foreground probability of the i-th negative sample proposal can be expressed as: According to the prospect probability p i , that is, the weight w of the negative samples can be recalculated so that the gradient of the background samples with lower similarity to the class prototype will not be significantly reduced, while the false negative samples with higher foreground probability are given lower weights. The weight calculation can be expressed as:

[0063] w=(1-p) γ exp(-λH(p));

[0064] H(p)=-plog(p)-(1-p)log(1-p);

[0065] Among them, γ is a regulatory factor used to control the decay rate of the weight of false negative samples. H(p) is information entropy, which is used to capture the uncertainty of probability. When p approaches 0 or 1, the entropy value is low, and the exponential decay term has little effect on the weight; when p is close to the middle value of 0.5, the entropy value is high, that is, the uncertainty of the sample as a foreground is greater, and the exponential decay term will effectively reduce the weight fluctuation. λ is a hyperparameter, which is the weight coefficient of entropy.

[0066] Further, selecting n negative samples from the negative samples as instance negative samples includes: bg Divided into true negative samples N t and false negative samples N f ; From the true negative samples N t and false negative samples N f Sample a group from the sample and combine them together as the instance negative sample N sample . Further, from the true negative samples N t and false negative samples N f A set of samples is sampled, including: for the true negative sample N t , IoU balanced sampling is used for sampling; for the false negative sample N f , random sampling was used for sampling.

[0067] Sample true negative samples and false negative samples respectively. For the false negative sample N f , extracted by random sampling. The true negative samples N with higher IoU t , is more likely to be a hard negative sample, and IoU balanced sampling is used to guide the distribution of hard negative samples. Specifically, consider t Sampling N sample According to the IoU range between the sample and the real box, it is divided into K intervals. Then the probability p of the sample being selected in the kth interval is k for: Among them, Ck represents the number of sampling candidates in the corresponding interval k. sample The negative samples are evenly distributed into each interval, and then samples are uniformly selected from these distributed intervals.

[0068] Furthermore, the negative sample N bg Divided into true negative samples N t and false negative samples N f , including: using the formula From the negative sample N bg Select the true negative sample N t , the rest are taken as false negative samples N f ; where g represents the annotation box, G represents the set of annotation boxes, iou(·,·) represents the standard intersection-over-union metric, and τ represents the maximum threshold for distinguishing true and false negative samples.

[0069] Among the negative samples obtained by region proposal assignment, both hard negative samples and false negative samples have class prototypes with high similarity, and the IoU between most potential positive samples and the true labels is less than 0.1. Therefore, an appropriate IoU threshold is set to distinguish the true negative samples N. t and false negative samples N f The differentiation process of each negative sample n can be expressed as:

[0070]

[0071] N bg represents the background proposal set obtained in the small sample incomplete annotation scenario, iou(·) represents the standard IoU metric, and τ represents the setting for distinguishing true and false negative samples.

[0072] Further, the latent variable z and the RoI feature are aggregated to obtain the fusion feature f, including: using f = F agg ([F1(q roi ⊙z k ),F2(q roi -z k ),q roi ]) The kth latent variable z k and the i-th RoI feature q roi Polymerization; F1, F2, F agg is a single independent fully connected layer, and ⊙ represents channel-level multiplication operation.

[0073] Furthermore, the difference between the target category c and the support feature S is used to represent the semantic consistency loss L sem , including: using the formula Calculate the cosine similarity between the semantic embedding e and the support feature S to represent the semantic consistency loss L sem , where eT is the transpose of the semantic embedding vector e, and ||·|| represents the paradigm.

[0074] Perform feature aggregation on support features and query features. Approximate class distribution N(μ k ,σ k ), from which the latent vector z is sampled k As the class center. A nonlinear fusion network is used to aggregate the sampled translation-invariant class prototype center and the RoI features obtained by the query branch. Different from direct multiplication fusion, element multiplication and subtraction are first performed here to improve their relationship with the new class features, and then they are concatenated with the original RoI features to reduce the noise caused by random sampling. The independent fully connected layers F1 and F2 are activated with the ReLU function and passed through the fully connected layer F agg Finally, feature aggregation is performed, for the class prototype z k and RoI feature q roi , the whole process can be expressed as follows:

[0075] f=F agg ([F1(q roi ⊙z k ),F2(q roi -z k ),q roi ]).

[0076] For the classification loss, the weight w obtained by the negative sample gradient optimization module is optimized, which can be expressed as:

[0077]

[0078] Where L pos and L neg Represents the classification loss of positive samples and negative samples respectively. m and n are the number of positive and negative samples respectively. Use cross entropy to predict the class and the classification target c i and Calculate. The optimization goal is to find the parameters θ of the optimal classifier. The gradient descent process can be expressed as:

[0079]

[0080] Where γ is the learning rate, when the negative sample is true, the sample weight w j Equal to 1, and when the negative sample is false, the sample weight can be tuned to reduce the weight of the false negative sample while maintaining the contribution of the hard negative sample to the model training.

[0081] Example 2

[0082] This embodiment provides an image detection method, including: if the to-be-detected category of the to-be-detected image set belongs to class M, the to-be-detected image set is input into a trained base class image detection model, a target detection image set is found therefrom, and the label and position of the target subject are marked.

[0083] Furthermore, the image detection method further comprises: if the to-be-identified category of the to-be-detected image set does not belong to the category M, using the few-sample support image set and the labeled query image D q ' and target category c' train the base image detection model until the total loss converges to obtain the target image detection model; the category to be identified belongs to the target category c'; the image set to be detected is input into the target image detection model, the target detection image set is found out, and the label and position of the target subject are marked.

[0084] Furthermore, the training process of the target image detection model is the same as the training process of the base class image detection model. The base class image detection model has the same structure as the original image detection model, including an aggregation module and a support branch, a query branch, and a detection head connected thereto; the support branch includes: a first backbone network and a variational autoencoder connected in sequence; the query branch includes: a second backbone network and a region generation network connected in sequence; the first backbone network and the second backbone network parameter twins; the training process of the target image detection model described below is based on the structure of the base class image detection model.

[0085] The following steps are performed during the training process: the support image set is input into the first backbone network for feature extraction to obtain the support feature S; the support feature S and the semantic embedding e of the target category c are input into the encoder of the variational autoencoder to generate the mean μ and variance σ, and the mean μ and variance σ are input into the decoder of the variational autoencoder to obtain the generated feature S'; the latent variable z = μ + σ is extracted from the variational autoencoder; the labeled query image D q Input the second backbone network for feature extraction to obtain query features, input the query features into the region generation network to obtain target proposals including positive samples and negative samples; perform RoIAlign operation on the target proposal to obtain RoI features; aggregate the latent variable z and RoI features to obtain fused features f; input the fused features f into the detection head to obtain the predicted category and prediction box The total classification loss L is expressed as the sum of the classification loss of positive samples and the classification loss of negative samples. cls , using the prediction box Calculate the regression loss L with the original label b reg , using the difference between the support feature S and the generated feature S' to represent the reconstruction loss L rec , calculate the KL loss L using the mean μ and variance σ KL; The semantic consistency loss L is expressed by using the difference between the semantic embedding e and the supporting feature S sem ; Using the formula L = L cls +L reg +L rec +m1·L KL +m2·L sem Calculate the total loss L; m1 and m2 are L KL and L sem Hyperparameters of .

[0086] Example 3

[0087] The present embodiment provides a base class training device for an image detection model, including: an initialization model for initializing an original image detection model, the original image detection model including an aggregation module and a support branch, a query branch and a detection head connected thereto; the support branch includes: a first backbone network and a variational autoencoder connected in sequence; the query branch includes: a second backbone network and a region generation network connected in sequence; the first backbone network and the second backbone network parameter twins; a training model for using a support image set, a labeled query image D q The original image detection model is trained with each target category c in class M until the total loss reaches convergence, and the base class image detection model is obtained;

[0088] The following steps are performed during the training process: input the support image set into the first backbone network for feature extraction to obtain the support feature S; input the support feature S and the semantic embedding e of the target category c into the encoder of the variational autoencoder to generate the mean μ and variance σ, and input the mean μ and variance σ into the decoder of the variational autoencoder to obtain the generated feature S'; extract the latent variable z = μ + σ from the variational autoencoder; input the labeled query image D q Input the second backbone network for feature extraction to obtain query features, input the query features into the region generation network to obtain target proposals including positive samples and negative samples; perform RoIAlign operation on the target proposal to obtain RoI features; aggregate the latent variable z and RoI features to obtain fused features f; input the fused features f into the detection head to obtain the predicted category and prediction box The total classification loss L is expressed as the sum of the classification loss of positive samples and the classification loss of negative samples. cls , using the prediction box Calculate the regression loss L with the original label b reg , using the difference between the support feature S and the generated feature S' to represent the reconstruction loss L rec , calculate the KL loss L using the mean μ and variance σ KL ; The semantic consistency loss L is expressed by using the difference between the semantic embedding e and the supporting feature S sem ; Using the formula L = Lcls +L reg +L rec +m1·L KL +m2·L sem Calculate the total loss L; m1 and m2 are L KL and L sem Hyperparameters of .

[0089] Among them, small sample target detection requires a small number of k shots to predict new classes. In the case of scarce data and large sample changes, it is difficult to represent the entire class distribution if only the support samples are encoded as a single feature vector. First, the support images can be enhanced. Specifically, the enhancement operations include color operations, geometric operations, etc. The selection of the enhancement operation is defined as a search problem, and its search space contains K sub-strategies. Each sub-strategy consists of N operations applied to a single image in sequence. During the training process, the image enhancement strategy is randomly selected and applied to the current image. The enhanced image and the original image are used as support images to extract features through the backbone network as the input of the Variational Autoencoder (VAE). VAE learns the probability distribution of data by embedding data into a latent space and reconstructing the input data from this space.

[0090] This embodiment adopts conditional variational prototype modeling (C-VAE). First, a pre-trained word embedding model is used to extract the semantic embedding of the class from the base class dataset as a conditional input. C-VAE combines the features of the support image with the semantic embedding of the class and maps it to the latent space. Through the C-VAE decoder, reconstructed features are generated. In the inference stage, the model uses the distribution learned by VAE on the base class data to estimate the distribution of the new class. Compared with the support features and reconstructed features of a specific class, the latent vector of the VAE intermediate process contains more general features of the class prototype. Therefore, a latent vector containing semantic information is sampled from the distribution and is feature aggregated with the query feature, so that the estimated new class center is closer to the true class center.

[0091] On the other hand, in the training sample images of FSOD, only the instances of the current target category are annotated, but in fact there may be multiple unannotated instances in the image, which will be regarded as negative samples (background). These false negative samples can mislead the classifier and fail to correctly distinguish the new class target and background. In order to reduce the impact of false negative samples, a method for optimizing negative samples is introduced in the query branch. First, the query sample is input into the region proposal network (RPN) to generate region proposals, which are assigned as positive samples (foreground) and negative samples (background). Among these negative samples, both hard negative samples and false negative samples have class prototypes with high similarity, and the two are distinguished by setting the IoU threshold. Given the negative sample proposal, RoI features are generated by RoI alignment pooling, and the cosine similarity between the RoI features and the class prototypes obtained from the support samples is calculated to obtain the foreground probability of the proposal, and the gradient of the negative sample proposal is weighted according to the foreground probability to reduce the weight of false negative samples while maintaining the contribution of hard negative samples to model training, so that the model can better generalize to new classes. Finally, the total loss is composed of the reconstruction loss in VAE, KL divergence, classification loss in the detection head, regression loss, and consistency loss.

[0092] Example 4

[0093] This embodiment provides an image detection device, including: if the to-be-detected category of the to-be-detected image set belongs to class M, the to-be-detected image set is input into a base class image detection model, a target detection image set is found therefrom, and a label and position of the target subject are marked.

[0094] Example 5

[0095] This embodiment provides an image processing device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a base class training method or an image detection method when executing the computer program.

[0096] Example 6

[0097] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the base class training method or the image detection method are implemented.

[0098] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A training method for a small sample image detection model, characterized in that: include: S1: Initialize the original image detection model, which includes an aggregation module and a support branch, a query branch and a detection head connected thereto; The support branch includes: a first backbone network and a variational autoencoder connected in sequence; the query branch includes: a second backbone network and a region generation network connected in sequence; the first backbone network and the second backbone network parameter twins; S2: Using the support image set and the labeled query image D q and each target category c in class M to train the original image detection model until the total loss reaches convergence, thereby obtaining a base class image detection model; The following steps are performed during the training process: the support image set is input into the first backbone network for feature extraction to obtain support features S; the support features S and the semantic embedding e of the target category c are input into the encoder of the variational autoencoder to generate mean μ and variance σ, and the mean μ and variance σ are input into the decoder of the variational autoencoder to obtain generated features S'; the latent variable z = μ + σ is extracted from the variational autoencoder; the labeled query image D q Input the second backbone network to extract features to obtain query features, input the query features to the region generation network to obtain target proposals including positive samples and negative samples; perform RoIAlign operation on the target proposal to obtain RoI features; aggregate the latent variable z and the RoI features to obtain fused features f; input the fused features f to the detection head to obtain a predicted category and prediction box Using prediction boxes Calculate the regression loss L with the original label b reg , using the difference between the support feature S and the generated feature S' to represent the reconstruction loss L rec , calculate the KL loss L using the mean μ and variance σ KL ; Using the difference between the semantic embedding e and the supporting feature S to represent the semantic consistency loss L sem ; Using the formula L = L cls +L reg +L rec +m1·L KL +m2·L sem Calculate the total loss L; m1 and m2 are L KL and L sem Hyperparameters of .

2. The training method for a small sample image detection model according to claim 1, characterized in that: The total classification loss L cls The calculation process is as follows: Select n negative samples from the negative samples as instance negative samples; use Calculate the RoI feature f of the jth instance negative sample j With the k-th class prototype S k The cosine similarity between use Calculate the foreground probability p of the jth instance negative sample j , M is the total number of class prototypes; using w j =(1-p j ) γ ·exp(-λH(p j )) Calculate the classification weight w of the jth instance negative sample j ; H(p j ) is p j The information entropy of is, λ is the hyperparameter, and γ is the adjustment factor; use Calculate the classification loss of negative samples; use L cls =L pos +L neg Calculate the total classification loss L cls , c j and are the true label and predicted label of the jth instance negative sample, L pos Classification loss for positive samples.

3. The training method for a small sample image detection model as claimed in claim 2, characterized in that: The step of selecting n negative samples from the negative samples as instance negative samples includes: The negative sample N bg Divided into true negative samples N t and false negative samples N f ; From the true negative samples N t And the false negative sample N f Sample a group from the sample and combine them together as the instance negative sample N sample .

4. The training method for a small sample image detection model as claimed in claim 3, characterized in that: The true negative samples N t And the false negative sample N f A set of samples is included: For the true negative sample N t , IoU balanced sampling is used for sampling; For the false negative sample N f , random sampling was used for sampling.

5. The training method for a small sample image detection model as claimed in claim 3, characterized in that: The negative sample N bg Divided into true negative samples N t and false negative samples N f , include: Using the formula From the negative sample N bg Select the true negative sample N t , the rest are taken as false negative samples N f ; where g represents the annotation box, G represents the set of annotation boxes, iou(·,·) represents the standard intersection-over-union metric, and τ represents the maximum threshold for distinguishing true and false negative samples.

6. The training method for a small sample image detection model according to claim 1, characterized in that: The step of aggregating the latent variable z and the RoI feature to obtain a fused feature f includes: Using f = F agg ([F1(q roi ⊙z k ),F2(q roi -z k ),q roi ]) The kth latent variable z k and the i-th RoI feature q roi Polymerization; F1, F2, F agg is a single independent fully connected layer, and ⊙ represents channel-level multiplication operation.

7. The training method for a small sample image detection model according to claim 1, characterized in that: The difference between the target category c and the support feature S is used to represent the semantic consistency loss L sem ,include: Using the formula Calculate the cosine similarity between the semantic embedding e and the support feature S to represent the semantic consistency loss L sem , where e T is the transpose of the semantic embedding vector e, and ||·|| represents the paradigm.

8. An image detection method, characterized in that: include: If the category to be identified of the image set to be detected belongs to class M, the image set to be detected is input into the base class image detection model trained by any one of claims 1-7, the target detection image set is found out, and the label and position of the target subject are marked.

9. The image detection method according to claim 8, characterized in that: Also includes: If the category to be identified in the image set to be detected does not belong to class M, then the few-sample support image set and the labeled query image D are used. q ' and target category c' to train the base image detection model until the total loss reaches convergence, thereby obtaining a target image detection model; the category to be identified belongs to target category c'; The image set to be detected is input into the target image detection model, a target detection image set is found therefrom, and the label and position of the target subject are marked.

10. The image detection method according to claim 9, characterized in that: The training process of the target image detection model is the same as the training process of the base class image detection model.

Images