Methods and apparatus for training object detection models

Abstract

This specification relates to a method and apparatus for training an object detection model. The method includes: acquiring a first object detection model trained on a first training set, wherein each image in the first training set belongs to K object categories; acquiring a second training set, wherein any image has a label category different from the K object categories; for any first image in the second training set, determining the superclass to which its label category belongs, and determining other object categories contained in the superclass category as the confusion category of the first image; the confusion category belongs to the K object categories; for the first image, determining a perturbation value that minimizes the objective function, thereby obtaining a first perturbation image to which the perturbation value is applied; determining a second extended training set based on each image in the second training set and its corresponding perturbation image, and training the first object detection model using the second extended training set to obtain a second object detection model.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of machine learning, and more particularly to methods and apparatus for training object detection models. Background Technology

[0002] Object detection is a fundamental task in computer vision, aiming to locate and classify different foreground objects in an image. Existing object detection models can achieve high recognition accuracy when trained on large-scale labeled data. However, large-scale labeled data is often difficult to collect and expensive. To address this issue, the industry has proposed Few-Shot Object Detection (FSOD), which aims to improve the performance of existing object detection models in detecting new object categories by utilizing a small number of labeled samples of new categories.

[0003] However, existing methods for training object detection models with limited sample sizes struggle to accurately learn the relationships and boundaries between different categories. This leads to the model misclassifying samples of new categories as existing ones, resulting in decreased detection accuracy. Therefore, a method is needed to improve the recognition accuracy of object detection models when detecting new categories of objects. Summary of the Invention

[0004] This specification describes one or more embodiments of a method and apparatus for training an object detection model. Based on the hierarchical structure of categories in prior knowledge, data augmentation is performed on sample images of new categories to generate training samples that are easily confused with existing categories, thereby enhancing the model's understanding of the relationship between new and old categories and improving the model's recognition accuracy for new categories.

[0005] Firstly, a method for training an object detection model is provided, including:

[0006] Obtain a first object detection model trained on a first training set, wherein each image in the first training set belongs to K object categories;

[0007] Obtain a second training set, in which any image has a label category different from the K object categories;

[0008] For any first image in the second training set, determine the superclass to which its label category belongs, and determine the other object categories contained in the superclass category as the confusion category of the first image; the confusion category belongs to the K object categories;

[0009] For the first image, a perturbation value that minimizes the objective function is determined, thereby obtaining a first perturbation image with the perturbation value applied; the objective function is the relative probability value of the predicted value for its label category and the predicted value for each confusion category obtained by inputting the first image with the added perturbation value into the first target detection model.

[0010] Based on each image in the second training set and its corresponding perturbation image, a second extended training set is determined, and the first target detection model is trained using the second extended training set to obtain the second target detection model.

[0011] In some possible implementations, determining the superclass to which the label category belongs, and identifying other object categories contained in the superclass as the obfuscated category of the first image, includes:

[0012] Obtain a pre-constructed tree structure diagram, which contains multiple object categories and the superclass category to which each object category belongs;

[0013] Based on the tree structure diagram, determine the superclass to which the label category of the first image belongs;

[0014] Based on the tree structure diagram, other object categories included in the superclass category are determined as the confusion categories of the first image.

[0015] In some possible implementations, the tree structure diagram is a three-level tree structure; wherein the root node represents all object categories, the second-level nodes contain each superclass category node, and the third-level nodes contain each object category node; any object category node is a child node of the superclass category node to which it belongs.

[0016] In some possible implementations, determining the perturbation value that minimizes the objective function includes:

[0017] Obtain the initial perturbation value;

[0018] Perform multiple iterations, and in each iteration, update the current perturbation value in the direction in which the objective function decreases.

[0019] In some possible implementations, obtaining the initial perturbation value includes:

[0020] A first random value matrix with the same size as the first image is obtained by sampling from a uniform distribution;

[0021] The first random value matrix is ​​multiplied by a preset first threshold to obtain the initial perturbation value.

[0022] In some possible implementations, the current perturbation value is updated, including:

[0023] The perturbation value is adjusted based on the objective function and the constraint domain using the projective gradient descent method; the constraint domain is determined based on the first image and the first threshold.

[0024] In some possible implementations, the first object detection model includes a classifier, which includes a processing layer and an output layer; the objective function is determined through the following steps:

[0025] Obtain the predicted values ​​for each object category obtained through the processing layer, and select the predicted values ​​for the label category and each obfuscated category from them;

[0026] The predicted values ​​of the label categories and each obfuscation category are input into the output layer, and the function value of the objective function is determined based on the probability values ​​output by the output layer.

[0027] In some possible implementations, the output layer includes a Softmax function; determining the function value of the objective function based on the probability value output by the output layer includes:

[0028] The logarithm of the probability value of the label category output by the Softmax function is determined as the function value of the objective function.

[0029] In some possible implementations, the label of the first perturbed image is the label of the first image.

[0030] In some possible implementations, determining the second extended training set includes:

[0031] For any current image in the second training set, determine whether the first object detection model can correctly identify the object category; if it can, add the current image and its corresponding perturbation image to the second extended training set; if it cannot, add only the current image to the second extended training set.

[0032] In some possible implementations, any perturbed image has the same label category as its corresponding current image.

[0033] In some possible implementations, a first number of images in the first training set have the same category label; a second number of images in the second training set have the same category label; the first number is greater than the second number, and the difference or ratio between the first number and the second number is greater than a preset threshold.

[0034] Secondly, an apparatus for training an object detection model is provided, comprising:

[0035] The first acquisition unit is configured to acquire a first object detection model trained based on a first training set, wherein each image in the first training set belongs to K object categories.

[0036] The second acquisition unit is configured to acquire a second training set, wherein any image has a label category different from the K object categories;

[0037] The confusion category determination unit is configured to, for any first image in the second training set, determine the superclass to which its label category belongs, and determine other object categories contained in the superclass as the confusion category of the first image; the confusion category belongs to the K object categories;

[0038] The perturbation image determination unit is configured to determine, for a first image, a perturbation value that minimizes the objective function, thereby obtaining a first perturbation image with the perturbation value applied; the objective function is the relative probability value of the predicted value for its label category and the predicted value for each confused category obtained by inputting the first image with the added perturbation value into a first target detection model.

[0039] The model training unit is configured to determine a second extended training set based on each image in the second training set and its corresponding perturbation image, and train the first object detection model using the second extended training set to obtain a second object detection model.

[0040] Thirdly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of the first aspect.

[0041] Fourthly, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method of the first aspect.

[0042] The method and apparatus for training an object detection model proposed in the embodiments of this specification are as follows: The method, based on prior knowledge, identifies other categories belonging to the same superclass as the new category image. Since these categories belong to the same superclass as the new category, they are more similar to each other than categories in different superclasses, and are therefore more likely to be misidentified and confused by the object detection model. Then, based on the idea of ​​adversarial training, perturbation noise is added to the new category image, making the perturbed image more easily identified by the original object detection model as another category within the same superclass, rather than another category in a different superclass. Next, based on the idea of ​​curriculum learning, the object detection model is trained from easy to difficult using original samples and perturbation samples. This allows it to explicitly learn the differences between the new category and other categories under the same superclass, thereby better distinguishing the new category from existing categories similar to it, and improving the accuracy of object detection. Attached Figure Description

[0043] To more clearly illustrate the technical solutions of the various embodiments disclosed in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only a few embodiments disclosed in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 A schematic diagram illustrating an implementation scenario of a method for training an object detection model according to one embodiment is shown.

[0045] Figure 2 A schematic diagram illustrating boundary changes based on a model according to one embodiment is shown;

[0046] Figure 3 A flowchart illustrating a method for training an object detection model according to one embodiment;

[0047] Figure 4 A schematic diagram of the process for generating a perturbation image according to one embodiment is shown;

[0048] Figure 5 A schematic block diagram of an apparatus for training an object detection model according to one embodiment is shown. Detailed Implementation

[0049] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0050] As mentioned earlier, in related technologies, the method of fine-tuning an existing target detection model based on a small number of new class samples (small sample target detection method) makes it difficult for the model to accurately learn the relationship and boundaries between different categories, causing the model to easily detect samples of new categories as existing categories, resulting in a decrease in detection accuracy.

[0051] After research, the inventors discovered that detection confusion in object detection models often occurs between categories within the same superclass. Specifically, object detection models are more inclined to incorrectly detect objects belonging to category A as belonging to categories B and C, which are also within the same superclass, rather than to belonging to categories D and E, which are different superclasses. A superclass is a set of categories that belong to a higher level than the categories themselves.

[0052] For example, consider a superclass "furniture," which includes categories like "chair," "table," and "sofa"; and another superclass "animals," which includes categories like "sheep" and "horse." Existing object detection models have already been trained on samples containing the categories "chair," "table," "sheep," and "horse." Now, we need to train the model using a small number of samples belonging to the new category "sofa." Since "sofa" and "chair" and "table" all belong to the superclass "furniture," they inherently possess high semantic similarity, making them more difficult to distinguish. However, "sofa" has very low similarity to "sheep" and "horse," which belong to the other superclass "animals," making it relatively easier to differentiate.

[0053] As a result, when existing object detection models are trained using a small number of samples belonging to the new category "sofa," the insufficient sample size and limitations of the technology mean that the models still struggle to accurately learn the clear distinctions between "sofa," "chair," and "table." This makes it easier for the trained object detection models to misidentify test samples belonging to the "sofa" category in the test set as "chair" or "table" instead of "sheep" or "horse" when errors occur.

[0054] This is because the categories belonging to the same superclass are more similar to other categories in different superclasses. When the number of training samples for a new category is insufficient, the object detection model has more difficulty learning the semantic space boundaries between the new category and other existing categories under the same superclass, which can easily lead to confusion between the categories in the same superclass.

[0055] Based on the above analysis, this specification proposes a method for training an object detection model to overcome the above problems. By constructing appropriate perturbation noise, the training samples of the new category are augmented, so that the trained object detection model can more clearly distinguish the semantic space boundaries between categories.

[0056] Figure 1 This diagram illustrates an implementation scenario of a method for training an object detection model according to one embodiment. Figure 1 In the example, the existing object detection model has been trained on training samples of categories 2, 3, and 5, while categories 1 and 4 are new categories to be learned. Category 1 belongs to the same superclass A as categories 2 and 3, and category 4 belongs to the same superclass B as categories 5. Figure 1 In the diagram, training samples of category 1 correspond to white circles, and training samples of category 4 correspond to dark gray circles.

[0057] For any training sample image 1 belonging to category 1, it has a corresponding label, i.e., category 1. First, determine the superclass to which category 1 belongs, i.e., superclass A. Then, determine the other categories in superclass A, i.e., category 2 and category 3, and define category 2 and category 3 as the confusion categories for image 1. Next, add perturbation noise to image 1, and use an object detection model to identify the noisy image 1. Adjust the noise value based on the recognition result so that the object detection model is more inclined to identify the noisy image 1 as a category within the confusion categories, i.e., category 2 or category 3. Finally, we obtain a perturbation image 1 with appropriate added noise. The specific method for adjusting the noise value will be described in detail in subsequent steps.

[0058] By setting the label of perturbation image 1 to the label of image 1, i.e., category 1, we obtain the enhanced sample corresponding to image 1, i.e., perturbation image 1 labeled with category 1. Since perturbation image 1 is generated based on image 1, but it is easily identified as a confused category, perturbation image 1 is essentially a sample generated based on the method in the embodiments of this specification, located near the boundary between category 1 and its confused category.

[0059] Using a similar approach, perturbation images can be generated for each training sample image in category 1. Training the object detection model using the perturbation images of category 1 allows it to better learn the difference between category 1 and its confused categories (category 2, category 3).

[0060] Similar to category 1, category 5 is designated as the confusion category for category 4. Perturbation images corresponding to each sample image of category 4 are then generated, and their labels are set to category 4. Training the object detection model using the perturbation images of category 4 allows it to better learn the difference between category 4 and its confusion category (category 5).

[0061] exist Figure 1 In the diagram, the perturbation image for category 1 corresponds to a light gray triangle, and the perturbation image for category 4 corresponds to a dark gray triangle.

[0062] The above describes the process of generating perturbed images; the following describes the process of training the object detection model.

[0063] The training of the object detection model consists of multiple rounds. In any round, a batch of samples is first sampled from the original training set containing samples of each new category. The perturbation image corresponding to each sample is determined according to its category. The perturbation image and the original image together form the current training set, which is used in this round of training.

[0064] During training, the object detection model is trained from easy to difficult, based on the concept of course learning. Specifically, for an original image in the current training set, the current object detection model is first used to detect objects. If the model can correctly identify the category of the original image, the training loss is calculated using the original image and the corresponding perturbation image. If the model cannot correctly identify the category of the original image, it should first correctly learn the relatively simple original image instead of directly learning the more complex perturbation image. In this case, only the original image is used to calculate the training loss.

[0065] like Figure 1 As shown, the dashed ellipse represents the samples used in the current training round. From Figure 1 As can be seen, as the training process progresses, the model's ability to recognize new categories continuously improves, and the number of perturbation images used during training also continuously increases.

[0066] After training the object detection model multiple times based on the aforementioned training process, the trained samples can learn more clearly the boundaries between new categories and existing categories, especially the boundaries between categories within the same superclass.

[0067] It should be noted that, Figure 1 The types of superclasses and categories, as well as the subsequent training process, are merely examples and do not constitute a limitation on the technical solutions of the embodiments in this specification.

[0068] The following is a specific example illustrating the beneficial effects of the methods described in the embodiments of this specification. Figure 2This diagram illustrates the identification of boundary changes based on a model according to one embodiment. Figure 2 As shown in (a), the left, right, and bottom sides of the box correspond to the semantic spaces of categories 1, 2, and 5, respectively. The solid line segments represent the boundaries between the three categories currently learned by the object detection model. Small white circles represent samples of category 1, light gray circles represent samples of category 2, and black circles represent samples of category 5. Categories 1 and 2 belong to the same superclass, while category 5 belongs to a different superclass. Based on the aforementioned analysis, the model should focus more on learning the boundary between categories 1 and 2.

[0069] according to Figure 2 As can be seen in (a), there is a sample of category 1 located in the semantic space of category 2, corresponding to the small white circle with thick lines. In other words, this sample will be incorrectly identified as category 2 by the object detection model.

[0070] The perturbation image corresponding to the sample image is generated by the method described in the embodiments of this specification. Figure 2 The small triangle in (b). The object detection model is trained using the perturbation image and the original sample image to obtain the updated boundary, as shown. Figure 2 As shown in (b). From Figure 2 As can be seen in (b), the boundary between category 1 and category 2 has moved from the original dashed line to the current solid line position. In other words, by training with perturbation images, the model is able to learn the boundary between the semantic spaces of category 1 and category 2 better.

[0071] Conversely, if randomly generated noise is added to the sample image, corresponding to Figure 2 The small triangle in (c). Using this perturbation image and the original sample image, a target detection model is trained to obtain the updated boundary, as shown below. Figure 2 As shown in (c). From Figure 2 As shown in (c), the randomly generated, less-than-ideal perturbation image fails to help the model better learn the boundary between category 1 and category 2. Instead, it alters the boundary it has learned between category 1 and category 5. Consequently, the model will still misclassify the samples corresponding to the small white circles with thick lines. Furthermore, the model's update of the boundary between category 1 and category 5 based on randomly generated perturbation image samples is not necessarily reasonable.

[0072] from Figure 2 As can be seen in the examples, the perturbation image samples generated in a directional manner according to the embodiments of this specification can help the model learn better the semantic space boundaries between categories under the same superclass, compared to randomly generated perturbation image samples.

[0073] The following describes the specific implementation steps of the above-described method for training the target detection model, with reference to specific embodiments.

[0074] Figure 3 A flowchart illustrating a method for training an object detection model according to one embodiment is provided. The method can be executed by any platform, server, or device cluster with computing and processing capabilities. Figure 3 As shown, the method includes at least the following steps: Step 302, obtaining a first object detection model trained on a first training set, wherein each image in the first training set belongs to K object categories; Step 304, obtaining a second training set, wherein any image has a label category different from the K object categories; Step 306, for any first image in the second training set, determining the superclass to which its label category belongs, and determining other object categories contained in the superclass category as the confusion category of the first image; the confusion category belongs to the K object categories; Step 308, for the first image, determining a perturbation value that minimizes the objective function, thereby obtaining a first perturbation image with the perturbation value applied; the objective function is the relative probability value of the predicted value for its label category and the predicted value for each confusion category obtained by inputting the first image with the added perturbation value into the first object detection model; Step 310, determining a second extended training set based on each image in the second training set and its corresponding perturbation image, and training the first object detection model using the second extended training set to obtain a second object detection model.

[0075] The specific execution process of each of the above steps is described below.

[0076] First, in step 302, a first object detection model trained based on a first training set is obtained, wherein each image in the first training set belongs to K object categories.

[0077] The first training set can be a basic training set, and the K object categories can be existing categories or simply basic categories. The first object detection model is pre-trained on the first training set and has the ability to detect objects in these K existing categories.

[0078] It should be noted that a single image sample can contain multiple foreground objects to be detected, and these foreground objects can belong to the same or different categories within the existing categories. In step 302, the images belonging to K object categories could mean that the foreground objects in the images belong to K object categories.

[0079] Then, in step 304, a second training set is obtained, wherein any image has a label category different from the K object categories.

[0080] The second training set can be a new category training set, used to train the first object detection model to detect objects of the new category. The label category, different from the K object categories, can be the new category. Any image sample in the second training set can contain multiple foreground objects to be detected, where foreground objects belonging to the new category will be labeled with the corresponding new category label. In step 304, any image having a label category different from the K object categories can mean that any foreground object in any image has a label category different from the K object categories.

[0081] In one embodiment, the number of sample images belonging to each new category in the second training set is much smaller than the number of sample images belonging to each existing category in the first training set, which is a training scenario for small sample object detection. In this embodiment, a first number of images in the first training set have the same category label; a second number of images in the second training set have the same category label; the first number is greater than the second number, and the difference or ratio between the first number and the second number is greater than a preset threshold.

[0082] When training the first object detection model using the new category training set, especially when the number of labeled training samples in the new category training set is small, the method in subsequent steps 306 to 310 can enable the object detection model to better learn the semantic space boundaries between each category, especially the semantic space boundaries between each category under the same superclass.

[0083] Next, in step 306, for any first image in the second training set, determine the superclass to which its label category belongs, and determine the other object categories contained in the superclass category as the confusion category of the first image; the confusion category belongs to the K object categories.

[0084] The confusing categories for a new category are the K object categories in the first training set that belong to the same superclass as the new category. Continuing the previous example, the confusing categories for the new category "sofa" could be the existing categories "table" and "chair," both of which belong to the superclass "furniture."

[0085] The confusion categories of each new category in the second training set can be predetermined and stored, and then the confusion category corresponding to the label category can be directly read when processing the first image in step 306.

[0086] The hierarchical relationship between each category and its superclass can be acquired in advance as prior knowledge, or it can be constructed in advance before training the object detection model.

[0087] In one embodiment, step 306 includes steps 3062 to 3066.

[0088] In step 3062, a pre-constructed tree structure diagram is obtained, which contains multiple object categories and the superclass category to which each object category belongs.

[0089] There are several methods for constructing tree structure graphs. For example, one can directly use a prior knowledge database containing hierarchical relationships between categories, such as WordNet, and then construct a tree structure graph based on the hierarchical relationships in the prior knowledge; alternatively, a large language model can be used, inputting the object categories from the first and second training sets into the large language model, which will then generate the tree structure graph. No specific limitations are imposed here.

[0090] In step 3064, the superclass to which the label category of the first image belongs is determined according to the tree structure diagram.

[0091] In step 3066, based on the tree structure diagram, other object categories included in the superclass category are determined as the confusion categories of the first image. Each of these other object categories belongs to one of the K object categories in the first training set.

[0092] Since the hierarchical relationship used in the embodiments of this specification is actually a two-level hierarchical relationship, the corresponding tree structure diagram can be a three-level tree structure.

[0093] In a more specific embodiment, the tree structure diagram is a three-level tree structure; wherein the root node represents all object categories, the second-level nodes contain each superclass category node, and the third-level nodes contain each object category node; any object category node is a child node of its superclass category node. An example of a tree structure diagram can be shown below. Figure 1 As shown on the left.

[0094] In other embodiments, in step 306, instead of explicitly constructing a tree structure graph, the confused categories of the label categories of the first image can be directly derived from prior knowledge. Alternatively, the confused categories of the label categories of the first image can be directly generated using a large language model; this is not a limitation.

[0095] Then, in step 308, for the first image, a perturbation value that minimizes the objective function is determined, thereby obtaining a first perturbation image with the perturbation value applied; the objective function is the relative probability value of the predicted value for its label category and the predicted value for each confusion category obtained by inputting the first image with the added perturbation value into the first object detection model.

[0096] The following combination Figure 4 Describe the specific implementation process of step 308. Figure 4 A schematic flowchart illustrating the generation of a perturbation image according to one embodiment is shown. It should be noted that... Figure 4This does not constitute a limitation on the embodiments described in this specification.

[0097] The first graph can be denoted as x, in Figure 4 The original image is represented by a two-dimensional matrix, with the corresponding label category denoted as y and the confusion category denoted as CC(x), which contains one or more categories from the first training set.

[0098] After applying a perturbation to the first image x, an initial perturbed image x′ is obtained, and the perturbed image x′ is input into the first target detection model for processing.

[0099] Typically, an object detection model includes a feature extraction network and a classifier. The feature extraction network, usually with a Convolutional Neural Network (CNN) as its backbone, is used to extract features from an image. The classifier includes processing layers for further processing of image features, and an output layer for classification. The processing layer may include one or more hidden layers, whose output is a predicted value z (or logits). The output layer receives the predicted value z and calculates the probability value p (or confidence score) of the foreground object in the input image belonging to each category based on the predicted value z.

[0100] In one embodiment, the first object detection model includes a classifier, which includes a processing layer and an output layer; the objective function in step 308 is determined through steps 3082 and 3084 as follows:

[0101] First, in step 3082, the predicted values ​​z for each object category obtained through the processing layer are obtained, and the predicted values ​​of the label category y and each confusion category CC(x) are selected to form a vector z′.

[0102] Specifically, the predicted value z output by the processing layer can be a vector with the same number of dimensions as the number of categories in the prior knowledge, including all categories from the first and second training sets. The values ​​of each dimension in the predicted value z correspond to the categories in the prior knowledge. The predicted values ​​of the label category y and each confusion category CC(x) are selected from this vector to form a vector z′, as shown below. Figure 4 As shown.

[0103] Then, in step 3084, the predicted values ​​z′ of the label category and each confusion category are input into the output layer, and the function value of the objective function is determined based on the probability value p′ output by the output layer.

[0104] As the foregoing analysis shows, object detection models are prone to confusion between categories within the same superclass. Step 3084 inputs only the predicted values ​​z′ of the label category y and each confusion category CC(x) into the output layer, so that when calculating the probability value, the output layer focuses on y and CC(x) without being affected by categories under other superclasses. The probability value p′ output by the output layer is the probability of the perturbation image with respect to the label category y and each confusion category CC(x).

[0105] Then, the value of the objective function can be determined based on the probability value p′.

[0106] In a more specific embodiment, the output layer includes a Softmax function. Step 3084, determining the function value of the objective function based on the probability values ​​output by the output layer, includes:

[0107] The probability value p of the label category y output by the Softmax function. y log(p′) y The value of the objective function is determined as shown in formula (1):

[0108]

[0109] Where log() is the natural logarithm, exp() is the natural exponent, and z′ y This represents the predicted value in the predicted value vector z′ corresponding to the label category y. i Let represent the value at the i-th position in z′, and k be the dimension of z′.

[0110] By reducing the probability value p of the first detection model for the perturbed image with respect to the label category y y By using this method, a more reasonable perturbation value can be obtained.

[0111] In other embodiments, the prediction loss of the first detection model can be calculated based on the probability value p′ and the label category y, using an existing loss function. Since the purpose of the loss function is to measure the difference between the model's prediction and the true label, specifically, the smaller the function value of the loss function, the closer the model's prediction is to the true label. However, the optimization objective of this embodiment is to make the model's prediction far from the true label and close to the confusion category CC(x). Therefore, in this embodiment, the function value of the objective function is the negative of the prediction loss.

[0112] After obtaining the objective function, the value of the perturbation can be optimized based on the objective function.

[0113] In one embodiment, step 308, which determines the perturbation value that minimizes the objective function, includes steps 3086 and 3088.

[0114] In step 3086, the initial perturbation value is obtained.

[0115] The disturbance value can be generated randomly, or it can be generated randomly under constraints.

[0116] In a more specific embodiment, obtaining the initial perturbation value includes:

[0117] A first random value matrix ξ with the same size as the first image is obtained by sampling from a uniform distribution;

[0118] Multiply the first random value matrix ξ by the preset first threshold ∈ to obtain the initial perturbation value ∈·ξ.

[0119] Then, in step 3088, multiple iterations are performed, and in each iteration, the current perturbation value is updated in the direction in which the objective function decreases.

[0120] The perturbation value ∈·ξ is applied to the first image x to obtain the initial perturbation image x′=x+∈·ξ. Then, based on the perturbation image x′ and the aforementioned steps 3082 and 3084, the value of the objective function is calculated.

[0121] In a more specific embodiment, step 3088, updating the current perturbation value, includes:

[0122] The perturbation value is adjusted based on the objective function and the constraint domain using the projection gradient descent method; the constraint domain is determined based on the first image x and the first threshold ∈.

[0123] The constraint domain of the first image x below the first threshold ∈ can be denoted as B. ∈ (x)={x′:‖xx′‖ m ≤∈}, where ‖x‖ m Let m be the m-norm of matrix x. Constraint domain B. ∈ (x) represents the set of all points whose distance to the first image is less than a first threshold, where the distance is measured using the m-norm. The m-norm can be a 1-norm, a 2-norm, an infinity norm, etc.

[0124] After determining the constraint domain, the perturbation value is adjusted based on the projective gradient descent method. In any update round, if the perturbation value exceeds the constraint domain B... ∈ If the range of (x) is defined, then project it back to the constraint domain B. ∈In (x), the purpose of the projection gradient descent method is to keep the perturbation of the first image x within a preset small range, that is, to "perturb" the first image x and avoid making too large a change to the first image x.

[0125] In other embodiments, the perturbation value can also be adjusted using the conventional gradient descent method. This is prior art and will not be described further here.

[0126] Since the purpose of step 308 is to make a small perturbation to the first image, the resulting first perturbation image x′ is still essentially in the label category y, but is easily identified as the confusing category CC(x).

[0127] Therefore, in one embodiment, the label of the first perturbed image x′ is the label y of the first image.

[0128] In step 308, the confidence score of the first object detection model for the first perturbation image x′ on the label y is reduced, while its confidence score on the confusion category CC(x) is increased. This makes the first object detection model more likely to identify the first perturbation image x′ as a category in the confusion category CC(x). Perturbation image samples labeled y are generated, but which are easily misidentified as CC(x). The sample corresponding to the first image can be (x,y), and the sample corresponding to the first perturbation image can be (x′,y).

[0129] Through a process similar to step 308, perturbation images corresponding to each image in the second training set can be generated. The process of generating perturbation images is essentially a form of data augmentation, which can effectively increase the number of training samples when the number of new category samples is insufficient.

[0130] Understandably, step 308 can be performed multiple times on the same image in the second training set to generate multiple perturbation images, further expanding the number of training samples.

[0131] Finally, in step 310, based on each image in the second training set and its corresponding perturbation image, a second extended training set is determined, and the first target detection model is trained using the second extended training set to obtain the second target detection model.

[0132] In one embodiment, determining the second extended training set includes:

[0133] For any current image in the second training set, determine whether the first object detection model can correctly identify the object category; if it can, add the current image and its corresponding perturbation image to the second extended training set; if it cannot, add only the current image to the second extended training set.

[0134] In this embodiment, step 310 can be any one of the multiple training rounds. In the current training round, the current object detection model is first used to identify a relatively simple current image. If it can identify it correctly, it means that it has learned the features in the image and can learn more complex perturbation images. If the current object detection model cannot correctly identify the current image, it will only learn the relatively simple current image in this training round and will not learn complex perturbation images.

[0135] In this embodiment, corresponding to step 308 above, any perturbation image has the same label category as its corresponding current image.

[0136] After obtaining the second extended training set, a corresponding loss function can be constructed based on each image sample and its label in the second extended training set to train the object detection model. This part is existing technology and will not be elaborated here.

[0137] In another embodiment, the second extended training set includes each image in the second training set and its corresponding perturbation image. In this embodiment, corresponding to step 308 above, any perturbation image has the same label category as its corresponding current image.

[0138] In this embodiment, training the first object detection model using the second extended training set includes:

[0139] For any current image in the second training set, determine whether the first object detection model can correctly identify the object category; if it can, calculate the training loss based on the current image and its corresponding perturbation image; if it cannot, calculate the training loss based only on the current image.

[0140] In step 310, based on the idea of ​​course learning, and combining the perturbation image generated in step 308 with the original images in the second training set, the first object detection model is trained from easy to difficult to obtain the second object detection model.

[0141] The method for training an object detection model provided in the embodiments of this specification is a feasible scheme for few-shot object detection. A tree structure graph is constructed using prior knowledge, thereby generating confusion categories for each new category. Based on the confusion categories, under the adversarial example generation mechanism, the generation of perturbation images for the confusion categories is guided. The labels of the perturbation images are the labels of the original images, but they are easily identified as other categories belonging to the same superclass as the categories described in the original images. This helps the object detection model to better learn the semantic space boundaries between categories under the same superclass.

[0142] Meanwhile, the embodiments in this specification, based on the characteristics of perturbed images and the concept of course learning, provide a method for training a target detection model that enables the model to learn from original image samples and perturbed image samples in a progressively more difficult manner.

[0143] Finally, the method for training the target detection model provided in the embodiments of this specification has wide applicability and can be extended to different small sample target detectors, all of which can achieve good results.

[0144] According to another embodiment, an apparatus for training an object detection model is also provided. Figure 5 A schematic block diagram of an apparatus for training an object detection model according to one embodiment is shown. This apparatus can be deployed in any device, platform, or cluster of devices with computing and processing capabilities. Figure 5 As shown, the device 500 includes:

[0145] The first acquisition unit 502 is configured to acquire a first object detection model trained based on a first training set, wherein each image in the first training set belongs to K object categories.

[0146] The second acquisition unit 504 is configured to acquire a second training set, wherein any image has a label category different from the K object categories;

[0147] The confusion category determination unit 506 is configured to, for any first image in the second training set, determine the superclass to which its label category belongs, and determine other object categories contained in the superclass category as the confusion category of the first image; the confusion category belongs to the K object categories;

[0148] The perturbation image determination unit 508 is configured to determine, for a first image, a perturbation value that minimizes the objective function, thereby obtaining a first perturbation image with the perturbation value applied; the objective function is the relative probability value of the predicted value for its label category and the predicted value for each confused category obtained by inputting the first image with the added perturbation value into a first target detection model.

[0149] The model training unit 510 is configured to determine a second extended training set based on each image in the second training set and its corresponding perturbation image, and train the first target detection model using the second extended training set to obtain a second target detection model.

[0150] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform the methods described in any of the above embodiments.

[0151] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method described in any of the above embodiments.

[0152] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0153] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0154] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0155] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0156] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for training an object detection model, comprising: Obtain a first object detection model trained on a first training set, wherein each image in the first training set belongs to K object categories; Obtain a second training set, in which any image has a label category different from the K object categories; For any first image in the second training set, determine the superclass to which its label category belongs, and determine the other object categories contained in the superclass category as the confusion category of the first image; the confusion category belongs to the K object categories; For the first image, determine the perturbation value that minimizes the objective function, thereby obtaining the first perturbation image with the perturbation value applied; The objective function is to input the first image with added perturbation values ​​into the first target detection model and obtain the relative probability values ​​of the predicted values ​​for its label category and the predicted values ​​for each confused category. Based on each image in the second training set and its corresponding perturbation image, a second extended training set is determined, and the first target detection model is trained using the second extended training set to obtain the second target detection model.

2. The method according to claim 1, determining the superclass to which its label category belongs, and determining other object categories included in the superclass as the confusion category of the first image, includes: Obtain a pre-constructed tree structure diagram, which contains multiple object categories and the superclass category to which each object category belongs; Based on the tree structure diagram, determine the superclass to which the label category of the first image belongs; Based on the tree structure diagram, other object categories included in the superclass category are determined as the confusion categories of the first image.

3. The method according to claim 2, wherein, The tree structure diagram is a three-level tree structure; the root node represents all object categories, the second-level nodes contain each superclass category node, and the third-level nodes contain each object category node; any object category node is a child node of its superclass category node.

4. The method according to claim 1, wherein determining the perturbation value that minimizes the objective function comprises: Obtain the initial perturbation value; Perform multiple iterations, and in each iteration, update the current perturbation value in the direction in which the objective function decreases.

5. The method according to claim 4, wherein obtaining the initial disturbance value includes: A first random value matrix with the same size as the first image is obtained by sampling from a uniform distribution; The first random value matrix is ​​multiplied by a preset first threshold to obtain the initial perturbation value.

6. The method according to claim 5, updating the current perturbation value, comprising: Based on the objective function and the constraint domain, the perturbation value is adjusted using the projective gradient descent method. The constraint domain is determined based on the first image and the first threshold.

7. The method according to claim 1, wherein, The first object detection model includes a classifier, which comprises a processing layer and an output layer; the objective function is determined through the following steps: Obtain the predicted values ​​for each object category obtained through the processing layer, and select the predicted values ​​for the label category and each obfuscated category from them; The predicted values ​​of the label categories and each obfuscation category are input into the output layer, and the function value of the objective function is determined based on the probability values ​​output by the output layer.

8. The method according to claim 7, wherein, The output layer includes a Softmax function; based on the probability values ​​output by the output layer, the function value of the objective function is determined, including: The logarithm of the probability value of the label category output by the Softmax function is determined as the function value of the objective function.

9. The method according to claim 1, wherein, The label of the first perturbation image is the label of the first image.

10. The method of claim 1, wherein determining the second extended training set comprises: For any current image in the second training set, determine whether the first object detection model can correctly identify the object category; If possible, add the current image and its corresponding perturbation image to the second extended training set; If not, simply add the current image to the second expanded training set.

11. The method according to claim 10, wherein, Any perturbation image has the same label category as its corresponding current image.

12. The method according to claim 1, wherein, A first number of images in the first training set have the same category label; a second number of images in the second training set have the same category label; the first number is greater than the second number, and the difference or ratio between the first number and the second number is greater than a preset threshold.

13. An apparatus for training an object detection model, comprising: The first acquisition unit is configured to acquire a first object detection model trained based on a first training set, wherein each image in the first training set belongs to K object categories. The second acquisition unit is configured to acquire a second training set, wherein any image has a label category different from the K object categories; The confusion category determination unit is configured to, for any first image in the second training set, determine the superclass to which its label category belongs, and determine other object categories contained in the superclass as the confusion category of the first image; the confusion category belongs to the K object categories; The perturbation image determination unit is configured to determine, for a first image, a perturbation value that minimizes the objective function, thereby obtaining a first perturbation image to which the perturbation value is applied; The objective function is to input the first image with added perturbation values ​​into the first target detection model and obtain the relative probability values ​​of the predicted values ​​for its label category and the predicted values ​​for each confused category. The model training unit is configured to determine a second extended training set based on each image in the second training set and its corresponding perturbation image, and train the first object detection model using the second extended training set to obtain a second object detection model.

14. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-12.

15. A computing device comprising a memory and a processor, wherein, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-12.

Images