Method and device for training target detection model

Abstract

The embodiment of the invention relates to a method and device for training a target detection model, and the method comprises the steps: obtaining a first target detection model obtained through training based on a first training set, and each image in the first training set belongs to K object types; obtaining a second training set, wherein any image has a label category different from K object categories; for any first image in the second training set, determining a super-class category to which the label category belongs, and determining other object categories included in the super-class category as confusion categories of the first image; the confusion category belongs to K object categories; aiming at the first image, determining a disturbance value which minimizes the target function so as to obtain a first disturbance image to which the disturbance value is applied; and based on each image in the second training set and the corresponding disturbance image, determining a second extended training set, and training the first target detection model by using the second extended training set to obtain a second target detection model.

Description

Technical Field

[0001] One or more embodiments of this specification relate to the field of machine learning, and in particular, to methods and devices for training target detection models. Background Art

[0002] Object detection is a basic task in the field of computer vision, which aims to locate and classify different foreground objects in an image. Existing object detection models can achieve high recognition accuracy when trained with large-scale annotated data. However, large-scale annotated data is usually difficult and expensive to collect. To solve this problem, the industry proposed Few-Shot Object Detection (FSOD), which aims to use a small number of annotated new category samples to train existing object detection models and improve their performance in detecting new category objects.

[0003] However, the existing methods for training small sample target detection models are difficult to accurately learn the relationship and boundaries between different categories, which makes it easy for the model to detect samples of new categories as existing categories, resulting in a decrease in detection accuracy. Therefore, a method is needed to improve the recognition accuracy of target detection models when detecting new categories of targets. Summary of the invention

[0004] One or more embodiments of the present specification describe a method and apparatus for training a target detection model, which performs data enhancement on sample images of new categories based on the category hierarchy in prior knowledge, and generates training samples that are easily confused with existing categories, so as to enhance the model's understanding of the relationship between new and old categories and improve the model's recognition accuracy for new categories.

[0005] In a first aspect, a method for training an object detection model is provided, comprising:

[0006] Obtaining a first object detection model trained based on a first training set, where each image in the first training set belongs to K object categories;

[0007] Obtain a second training set, wherein any image has a label category different from the K object categories;

[0008] For any first image in the second training set, determine the superclass category to which its label category belongs, and determine other object categories included in the superclass category as confusion categories of the first image; the confusion category belongs to the K object categories;

[0009] For the first image, a disturbance value that minimizes an objective function is determined, thereby obtaining a first disturbed image to which the disturbance value is applied; the objective function is a relative probability value between a prediction value for its label category and a prediction value for each confusion category obtained by inputting the first image to which the disturbance value is added into a first object detection model;

[0010] Based on each image in the second training set and its corresponding perturbation image, a second extended training set is determined, and the first object detection model is trained by using the second extended training set to obtain a second object detection model.

[0011] In some possible implementations, determining the superclass category to which the label category belongs, and determining other object categories included in the superclass category as the confusion category of the first image includes:

[0012] Acquire a pre-constructed tree structure diagram, wherein the tree structure diagram includes a plurality of object categories and a super-category category to which each object category belongs;

[0013] Determine, according to the tree structure diagram, the superclass category to which the label category of the first image belongs;

[0014] According to the tree structure diagram, other object categories included in the super category are determined as confusion categories of the first image.

[0015] In some possible implementations, the tree structure diagram is a three-layer tree structure; wherein the root node represents all object categories, the second-layer nodes include each super-class category node, and the third-layer nodes include each object category node; any object category node is a child node of the super-class category node to which it belongs.

[0016] In some possible implementations, determining a disturbance value that minimizes the objective function includes:

[0017] Get the initial disturbance value;

[0018] Multiple rounds of iterations are performed, and in each round of iterations, the current disturbance value is updated in the direction in which the objective function decreases.

[0019] In some possible implementations, obtaining an initial disturbance value includes:

[0020] Sampling from a uniform distribution to obtain a first random value matrix having the same size as the first image;

[0021] The first random value matrix is ​​multiplied by a preset first threshold to obtain the initial disturbance value.

[0022] In some possible implementations, updating the current disturbance value includes:

[0023] According to the objective function and the constraint domain, the disturbance value is adjusted based on a projected gradient descent method; the constraint domain is determined according to the first image and the first threshold.

[0024] In some possible implementations, the first target detection model includes a classifier, the classifier includes a processing layer and an output layer; and the objective function is determined by the following steps:

[0025] Obtaining the predicted values ​​of each object category obtained by the processing layer, and selecting the label category and the predicted values ​​of each confusion category therefrom;

[0026] The label category and the predicted values ​​of each confusion category are input into the output layer, and the function value of the objective function is determined according to the probability value output by the output layer.

[0027] In some possible implementations, the output layer includes a Softmax function; and determining a function value of the objective function according to a probability value output by the output layer includes:

[0028] The logarithmic value of the probability value of the label category output by the Softmax function is determined as the function value of the objective function.

[0029] In some possible implementations, the label of the first disturbed image is the label of the first image.

[0030] In some possible implementations, determining the second extended training set includes:

[0031] For any current image in the second training set, determine whether the first target detection model can correctly identify the object category therein; if so, add both the current image and its corresponding perturbation image to the second extended training set; if not, only add the current image to the second extended training set.

[0032] In some possible implementations, any perturbed image and its corresponding current image have the same label category.

[0033] In some possible implementations, a first number of images in the first training set have the same category label; a second number of images in the second training set have the same category label; the first number is greater than the second number, and a difference or ratio between the first number and the second number is greater than a preset threshold.

[0034] In a second aspect, a device for training a target detection model is provided, comprising:

[0035] A first acquisition unit is configured to acquire a first target detection model trained based on a first training set, wherein each image in the first training set belongs to K object categories;

[0036] A second acquisition unit is configured to acquire a second training set, wherein any image has a label category different from the K object categories;

[0037] a confusion category determination unit configured to, for any first image in the second training set, determine a super category to which its label category belongs, and determine other object categories included in the super category as the confusion category of the first image; the confusion category belongs to the K object categories;

[0038] A perturbed image determination unit is configured to determine, for the first image, a perturbation value that minimizes an objective function, thereby obtaining a first perturbed image to which the perturbation value is applied; the objective function is a relative probability value between a prediction value for its label category and a prediction value for each confusion category obtained by inputting the first image to which the perturbation value is added into the first object detection model;

[0039] The model training unit is configured to determine a second extended training set based on each image in the second training set and its corresponding perturbation image, and use the second extended training set to train the first target detection model to obtain a second target detection model.

[0040] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method of the first aspect.

[0041] In a fourth aspect, a computing device is provided, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method of the first aspect is implemented.

[0042] The method and device for training a target detection model proposed in the embodiments of this specification, the method is based on prior knowledge, and determines other categories that belong to the same superclass category as the category of the new category image. Since these categories belong to the same superclass as the new category, they are more similar to each other than categories of different superclasses, and are more likely to be misidentified and confused by the target detection model. Then, based on the idea of ​​adversarial training, perturbation noise is added to the new category image, so that the perturbed image with the perturbation noise added is more easily recognized by the original target detection model as other categories in the same superclass category, rather than other categories of different superclass categories. Next, based on the idea of ​​curriculum learning, by using original samples and perturbation samples, the target detection model is trained from easy to difficult, so that it can explicitly learn the difference between the new category and other categories under the same superclass category, so as to better distinguish the new category from existing categories similar to it, so as to improve the accuracy of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the multiple embodiments disclosed in this specification, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only the multiple embodiments disclosed in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A schematic diagram showing an implementation scenario of a method for training a target detection model according to an embodiment;

[0045] Figure 2 A schematic diagram showing a model identifying boundary changes according to one embodiment;

[0046] Figure 3 A flowchart showing a method for training an object detection model according to one embodiment;

[0047] Figure 4 A schematic diagram of a process for generating a disturbance image according to an embodiment is shown;

[0048] Figure 5 A schematic block diagram of an apparatus for training a target detection model according to an embodiment is shown. DETAILED DESCRIPTION

[0049] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0050] As mentioned above, in the related art, the method of fine-tuning the existing target detection model based on a small number of new category samples (small sample target detection method) makes it difficult for the model to accurately learn the relationship and boundaries between different categories, causing the model to easily detect samples of new categories as existing categories, resulting in a decrease in detection accuracy.

[0051] After research, the inventors found that the detection confusion of the target detection model often occurs in the categories within the same super-class category. Specifically, the target detection model is more likely to mistakenly detect the target belonging to category A as category B or category C, which belong to the same super-class category as category A, rather than category D or category E, which belong to different super-class categories. A super-class refers to a set of categories that are one level higher than a category and to which multiple categories belong together.

[0052] For example, one superclass is "furniture", which includes the categories "chair", "table", and "sofa"; another superclass is "animal", which includes the categories "sheep" and "horse". The existing object detection model has been trained on samples including the categories "chair", "table", "sheep", and "horse", and now it is necessary to continue training the object detection model with a small number of samples belonging to the new category "sofa". Since "sofa" and "chair" and "table" all belong to the superclass "furniture", they have a high degree of intrinsic semantic similarity and are more difficult to distinguish. Compared with "sheep" and "horse" belonging to the other superclass "animal", "sofa" has a low similarity and is relatively easier to distinguish.

[0053] As a result, when the relevant technology uses a small number of samples belonging to the new category "sofa" to train the existing object detection model, due to the insufficient number of samples and the limitations of the relevant technology, the object detection model still finds it difficult to accurately learn the clear boundaries between "sofa" and "chair" and "table". This makes it easier for the trained object detection model to mistakenly identify the test samples belonging to the "sofa" category in the test set as "chair" or "table" instead of "sheep" or "horse" when an identification error occurs.

[0054] This is because the categories belonging to the same superclass are more similar than other categories in different superclasses. When the number of training samples for a new category is insufficient, it is more difficult for the target detection model to learn the boundary of the semantic space between the new category and other existing categories under the same superclass, which makes it easy to confuse the categories in the same superclass.

[0055] Based on the above analysis, the embodiment of this specification proposes a method for training a target detection model to overcome the above problems. By constructing appropriate perturbation noise, sample enhancement is performed on new category training samples, so that the trained target detection model can more clearly distinguish the boundaries of the semantic space between categories.

[0056] Figure 1 A schematic diagram of an implementation scenario of a method for training a target detection model according to an embodiment is shown. Figure 1 In the example, the existing object detection model has been trained on training samples of category 2, category 3, and category 5, while category 1 and category 4 are new categories to be learned. Category 1, category 2, and category 3 belong to the same superclass category A, and category 4 and category 5 belong to the same superclass category B. Figure 1 In the figure, the training samples of category 1 correspond to white circles, and the training samples of category 4 correspond to dark gray circles.

[0057] For any training sample image 1 belonging to category 1, it has a corresponding label, namely category 1. First, determine the superclass category to which category 1 belongs, namely superclass A, and then determine other categories in superclass A, namely category 2 and category 3, and determine category 2 and category 3 as the confusion category of image 1. Then, add perturbation noise to image 1, and use the target detection model to recognize the image 1 with the added noise, and adjust the noise value according to the recognition result, so that the target detection model is more inclined to recognize the image 1 with the added noise as a category in the confusion category, namely category 2 or category 3. Finally, a perturbed image 1 with appropriate noise added is obtained. The specific method of adjusting the noise value will be described in detail in the subsequent steps.

[0058] The label of the perturbed image 1 is set to the label of image 1, that is, category 1, and the enhanced sample corresponding to image 1 is obtained, that is, the perturbed image 1 with category 1 as the label. Since the perturbed image 1 is generated based on image 1, but it is easily identified as a confused category, the perturbed image 1 is essentially a sample located near the boundary between category 1 and its confused category, which is generated based on the method in the embodiments of this specification.

[0059] In a similar manner, perturbed images corresponding to each training sample image in category 1 can be generated. Using perturbed images of category 1 to train the target detection model can enable it to better learn the difference between category 1 and its confusing categories (category 2, category 3).

[0060] Similar to category 1, the confusion category of category 4 is determined to be category 5, and then the perturbed images corresponding to each sample image of category 4 are generated, and their labels are set to category 4. Using the perturbed images of category 4 to train the object detection model can enable it to better learn the difference between category 4 and its confusion category (category 5).

[0061] exist Figure 1 In the figure, the perturbation images of category 1 correspond to light gray triangles, and the perturbation images of category 4 correspond to dark gray triangles.

[0062] The above describes the process of generating perturbed images. The following describes the process of training the target detection model.

[0063] The training of the object detection model consists of multiple rounds of training. In any round of training, a batch of samples are first sampled from the original training set containing samples of each new category, and the corresponding perturbation images are determined according to the category of each sample. The perturbed images and the original images together form the current training set, which is used in this round of training.

[0064] During training, based on the idea of ​​curriculum learning, the object detection model is trained from easy to difficult. Specifically, for an original image in the current training set, the current object detection model is first used to detect the object. If the model can correctly identify the category of the original image, the original image is used together with the corresponding perturbation image to calculate the training loss; if the model cannot correctly identify the category of the original image, the model should first correctly learn this relatively simple original image instead of directly learning the more complex perturbation image. At this time, only the original image is used to calculate the training loss.

[0065] like Figure 1 As shown in , the dotted ellipse represents the samples used in the current round of training. Figure 1 As can be seen in the figure, as the training process progresses, the model's ability to recognize new categories continues to increase, and the number of perturbation images used in training continues to increase.

[0066] After multiple rounds of training of the target detection model based on the aforementioned training process, the trained samples can more clearly learn the boundaries between new categories and existing categories, especially the boundaries between categories in the same supercategory.

[0067] It should be noted that Figure 1 The types of superclasses and categories, as well as the subsequent training process, are merely examples and do not limit the technical solutions of the embodiments of this specification.

[0068] A specific example is given below to describe the beneficial effects of the method of the embodiments of this specification. Figure 2FIG. 2 is a schematic diagram showing a model identifying boundary changes according to an embodiment. Figure 2 As shown in (a), the left, right, and bottom sides of the box correspond to the semantic spaces of category 1, category 2, and category 5, respectively. The solid line segments represent the boundaries between the three categories currently learned by the target detection model. The white small circles represent samples of category 1, the light gray small circles represent samples of category 2, and the black small circles represent samples of category 5. Among them, category 1 and category 2 belong to the same superclass category, and category 5 belongs to another superclass category. According to the above analysis, the model should focus more on learning the boundary between category 1 and category 2.

[0069] according to Figure 2 As can be seen in (a), there is a sample of category 1 located in the semantic space of category 2, corresponding to the small white circle with a thick line, which means that the sample will be mistakenly identified as category 2 by the target detection model.

[0070] The disturbed image corresponding to the sample image is generated by the method of the embodiment of this specification, corresponding to Figure 2 The small triangle in (b) of . Use the perturbed image and the original sample image to train the target detection model and get the updated boundary, such as Figure 2 As shown in (b). Figure 2 In (b), we can see that the boundary between category 1 and category 2 has moved from the original dotted line to the current solid line position. That is to say, through training with perturbed images, the model can better learn the boundary between the semantic spaces of category 1 and category 2.

[0071] On the contrary, if randomly generated noise is added to the sample image, corresponding to Figure 2 The small triangle in (c) of . Use the perturbed image and the original sample image to train the target detection model and obtain the updated boundary, as shown in Figure 2 As shown in (c). Figure 2 As can be seen in (c), the randomly generated perturbation image, which is not ideal, cannot help the model learn the boundary between category 1 and category 2 better. Instead, it changes the boundary between category 1 and category 5 that the model has learned. As a result, the model will still misclassify the samples corresponding to the small white circles with thick lines. Moreover, the model's update of the boundary between category 1 and category 5 based on the randomly generated perturbation image samples is not necessarily reasonable.

[0072] from Figure 2 As can be seen from the example, the perturbed image samples generated directionally according to the method of the embodiments of this specification can help the model better learn the boundaries of the semantic space between categories under the same superclass category compared to randomly generated perturbed image samples.

[0073] The specific implementation steps of the above-mentioned method for training the target detection model are described below in conjunction with specific embodiments.

[0074] Figure 3 A flowchart of a method for training a target detection model according to an embodiment is shown. The execution subject of the method can be any platform or server or device cluster with computing and processing capabilities. Figure 3 As shown, the method at least includes: step 302, obtaining a first target detection model trained based on a first training set, wherein each image in the first training set belongs to K object categories; step 304, obtaining a second training set, wherein any image has a label category different from the K object categories; step 306, for any first image in the second training set, determining the superclass category to which its label category belongs, and determining other object categories contained in the superclass category as the confusion category of the first image; the confusion category belongs to the K object categories; step 308, for the first image, determining a perturbation value that minimizes an objective function, thereby obtaining a first perturbed image to which the perturbation value is applied; the objective function is, when the first image to which the perturbation value is added is input into the first target detection model, the relative probability value of the predicted value for its label category and the predicted value for each confusion category obtained; step 310, determining a second extended training set based on each image in the second training set and its corresponding perturbed image, and using the second extended training set to train the first target detection model to obtain a second target detection model.

[0075] The specific execution process of each of the above steps is described below.

[0076] First, in step 302, a first object detection model trained based on a first training set is obtained, where each image in the first training set belongs to K object categories.

[0077] The first training set may be a basic training set, and the K object categories may be existing categories, or may also be referred to as basic categories. The first target detection model is pre-trained on the first training set and has the ability to perform target detection on the K existing categories.

[0078] It should be noted that a single image sample may contain multiple foreground objects to be detected, and each foreground object may belong to the same or different categories in the existing categories. In step 302, each image belongs to K object categories, which means that the foreground objects in the image belong to K object categories.

[0079] Then, in step 304, a second training set is obtained, in which any image has a label category different from the K object categories.

[0080] The second training set may be a new category training set, which is used to train the first target detection model so that it can detect targets of new categories. The label category different from the K object categories may be a new category. Any image sample in the second training set may contain multiple foreground targets to be detected, wherein the foreground targets belonging to the new category will be marked with the corresponding new category label. Any image in step 304 having a label category different from the K object categories may be that the foreground target in any image has a label category different from the K object categories.

[0081] In one embodiment, the number of sample images belonging to each new category in the second training set is much smaller than the number of sample images belonging to each existing category in the first training set, which is a training scenario for small sample target detection. In this embodiment, a first number of images in the first training set have the same category label; a second number of images in the second training set have the same category label; the first number is greater than the second number, and the difference or ratio between the first number and the second number is greater than a preset threshold.

[0082] When using the new category training set to train the first target detection model, especially when the number of labeled training samples in the new category training set is small, based on the methods of subsequent steps 306 to 310, the target detection model can better learn the boundaries of the semantic space between each category, especially the boundaries of the semantic space between each category under the same super category.

[0083] Next, in step 306, for any first image in the second training set, the superclass category to which its label category belongs is determined, and other object categories included in the superclass category are determined as confusion categories of the first image; the confusion categories belong to the K object categories.

[0084] The confusion categories of a new category are the categories of the K object categories in the first training set that belong to the same supercategory as the new category. Continuing with the previous example, the confusion categories of the new category "sofa" can be "table" and "chair" in the existing categories, both of which belong to the supercategory "furniture".

[0085] The confusion category of each new category in the second training set may be predetermined and stored, and then the confusion category corresponding to its label category may be directly read when processing the first image in step 306 .

[0086] The hierarchical relationship between each category and the supercategory can be pre-acquired as prior knowledge or pre-constructed before training the object detection model.

[0087] In one embodiment, step 306 includes steps 3062 to 3066 .

[0088] In step 3062, a pre-constructed tree structure diagram is obtained, wherein the tree structure diagram includes a plurality of object categories and a super-category category to which each object category belongs.

[0089] There are many ways to construct a tree structure diagram. For example, you can directly use a priori knowledge database containing a hierarchical relationship of categories, such as WordNet, and then construct a tree structure diagram based on the hierarchical relationship in the prior knowledge; or you can use a large language model, input each object category in the first training set and the second training set into the large language model, and let the large language model generate a tree structure diagram. There is no limitation here.

[0090] In step 3064, the superclass category to which the label category of the first image belongs is determined according to the tree structure diagram.

[0091] In step 3066, other object categories included in the super-category are determined according to the tree structure diagram as confusion categories of the first image, wherein each of the other object categories belongs to the K object categories in the first training set.

[0092] Since the hierarchical relationship used in the embodiments of this specification is essentially a two-layer hierarchical relationship, the corresponding tree structure diagram can be a three-layer tree.

[0093] In a more specific embodiment, the tree structure diagram is a three-layer tree structure; the root node represents all object categories, the second-layer nodes include each super-class category node, and the third-layer nodes include each object category node; any object category node is a child node of the super-class category node to which it belongs. An example of a tree structure diagram can be as follows: Figure 1 as shown on the left.

[0094] In other embodiments, the tree structure diagram may not be explicitly constructed in step 306, but the confusion category of the label category of the first image may be directly generated from the prior knowledge. Alternatively, the confusion category of the label category of the first image may be directly generated using a large language model, which is not limited here.

[0095] Then, in step 308, a disturbance value that minimizes an objective function is determined for the first image, thereby obtaining a first perturbed image to which the disturbance value is applied; the objective function is inputting the first image to which the perturbation value is added into the first target detection model, and obtaining a relative probability value of a prediction value for its label category and a prediction value for each confusion category.

[0096] The following combination Figure 4 , describing the specific implementation process of step 308. Figure 4 FIG. 1 is a flow chart of generating a disturbance image according to an embodiment. It should be noted that: Figure 4It does not constitute a limitation on the embodiments of this specification.

[0097] The first image can be denoted as x. Figure 4 In corresponds to the original image, which is a two-dimensional matrix, the corresponding label category can be recorded as y, and its confusion category can be recorded as CC(x), which contains one or more categories in the first training set.

[0098] After applying perturbation to the first image x, an initial perturbation image x′ is obtained, and the perturbation image x′ is input into the first object detection model for processing.

[0099] Generally speaking, an object detection model can include a feature extraction network and a classifier. The feature extraction network usually uses a convolutional neural network (CNN) as the main part to extract image features. The classifier includes a processing layer for further processing image features and an output layer for classification. The processing layer can include one or more hidden layers, and its output is a predicted value z (or logical value, logits); the output layer receives the predicted value z and calculates the probability value p (or confidence score) of the foreground target in the input image belonging to each category based on the predicted value z.

[0100] In one embodiment, the first target detection model includes a classifier, and the classifier includes a processing layer and an output layer; the objective function in step 308 is determined by the following steps 3082 and 3084:

[0101] First, in step 3082, the predicted value z of each object category obtained by the processing layer is obtained, and the predicted value of the label category y and each confusion category CC(x) is selected therefrom to form a vector z′.

[0102] Specifically, the predicted value z output by the processing layer can be a vector whose dimension is the same as the number of categories in the prior knowledge, including each category in the first training set and the second training set. The values ​​of each dimension in the predicted value z correspond to each category in the prior knowledge, from which the predicted values ​​of the label category y and each confusion category CC(x) are selected to form a vector z′, such as Figure 4 shown.

[0103] Then, in step 3084, the label category and the predicted value z′ of each confusion category are input into the output layer, and the function value of the objective function is determined according to the probability value p′ output by the output layer.

[0104] According to the above analysis, the target detection model is prone to confusion between categories under the same superclass. Step 3084 only inputs the predicted value z' of the label category y and each confused category CC(x) into the output layer, so that when the output layer calculates the probability value, it will focus on y and CC(x) without being affected by the categories under other superclasses. The probability value p' output by the output layer is the probability of the perturbed image with respect to the label category y and each confused category CC(x).

[0105] Then, the function value of the objective function can be determined according to the probability value p′.

[0106] In a more specific embodiment, the output layer includes a Softmax function. Step 3084 determines the function value of the objective function according to the probability value output by the output layer, including:

[0107] The probability value p of the label category y output by the Softmax function y The logarithm of ′log(p′ y ), is determined as the function value of the objective function. As shown in formula (1):

[0108]

[0109] Where log() is the natural logarithm, exp() is the natural exponential, and z′ y Represents the predicted value corresponding to the label category y in the predicted value vector z′, z′ i Represents the value of the i-th position in z′, and k is the dimension of z′.

[0110] By reducing the probability value p of the first detection model for the perturbed image about the label category y y ′, a more reasonable disturbance value can be obtained.

[0111] In other embodiments, the prediction loss of the first detection model can also be calculated based on the existing loss function according to the probability value p′ and the label category y. Since the purpose of the loss function is to measure the difference between the prediction result of the model and the true label, specifically, the smaller the function value of the loss function, the closer the prediction result of the model is to the true label. However, the optimization goal of the embodiment of this specification is to make the prediction result of the model away from the true label and close to the confusion category CC(x). Therefore, in this embodiment, the function value of the objective function is the opposite of the prediction loss.

[0112] After obtaining the objective function, the value of the disturbance can be optimized according to the objective function.

[0113] In one embodiment, determining the disturbance value that minimizes the objective function in step 308 includes step 3086 and step 3088 .

[0114] In step 3086, an initial disturbance value is obtained.

[0115] The disturbance value can be generated randomly, or some constraints can be added and randomly generated under the constraints.

[0116] In a more specific embodiment, obtaining an initial disturbance value includes:

[0117] Sampling from a uniform distribution to obtain a first random value matrix ξ having the same size as the first image;

[0118] The first random value matrix ξ is multiplied by a preset first threshold ∈ to obtain the initial disturbance value ∈·ξ.

[0119] Then, in step 3088, multiple rounds of iterations are performed, and in each round of iterations, the current disturbance value is updated in the direction in which the objective function decreases.

[0120] The disturbance value ∈·ξ is applied to the first image x to obtain an initial disturbance image x′=x+∈·ξ. Then, the value of the objective function is calculated based on the disturbance image x′ and the aforementioned steps 3082 and 3084.

[0121] In a more specific embodiment, updating the current disturbance value in step 3088 includes:

[0122] According to the objective function and the constraint domain, the disturbance value is adjusted based on the projected gradient descent method; the constraint domain is determined according to the first image x and the first threshold ∈.

[0123] The constraint domain of the first image x below the first threshold ∈ can be denoted as B ∈ (x)={x′:‖xx′‖ m ≤∈}, where ‖x‖ m Represents the m-norm of matrix x. Constraint domain B ∈ (x) represents a set of all points whose distance from the first image is less than a first threshold, where the distance is measured using the m-norm, which can be the first norm, the second norm, the infinite norm, and so on.

[0124] After the constraint domain is determined, the perturbation value is adjusted based on the projected gradient descent method. In any round of update, if the perturbation value exceeds the constraint domain B ∈ (x), then project it back to the constraint domain B. ∈(x). The function of the projected gradient descent method is to always keep the disturbance of the first image x within a preset small range, that is, to "perturb" the first image x to avoid excessive changes to the first image x.

[0125] In other embodiments, the disturbance value may also be adjusted using a common gradient descent method, which is a prior art and will not be described in detail here.

[0126] Since the purpose of step 308 is to perform only a slight perturbation on the first image, the obtained first perturbation image x′ essentially still belongs to the label category y, but is easily identified as the confusion category CC(x).

[0127] Therefore, in one embodiment, the label of the first disturbed image x′ is the label y of the first image.

[0128] Through step 308, the confidence score of the first object detection model for the first perturbed image x' on the label y is reduced, while its confidence score on the confusion category CC(x) is increased. The first object detection model tends to identify the first perturbed image x' as a category in the confusion category CC(x). This is used to generate a perturbed image sample with a label of y but easily mistakenly identified as CC(x). The sample corresponding to the first image may be (x, y), and the sample corresponding to the first perturbed image may be (x', y).

[0129] The perturbed images corresponding to the images in the second training set can be generated through a process similar to step 308. The process of generating perturbed images is essentially a kind of data augmentation, which can effectively expand the number of training samples when the number of new category samples is insufficient.

[0130] It is understandable that step 308 may be performed multiple times for the same image in the second training set to generate multiple perturbed images, thereby further expanding the number of training samples.

[0131] Finally, in step 310, based on each image in the second training set and its corresponding perturbation image, a second extended training set is determined, and the first object detection model is trained using the second extended training set to obtain a second object detection model.

[0132] In one embodiment, determining the second extended training set includes:

[0133] For any current image in the second training set, determine whether the first target detection model can correctly identify the object category therein; if so, add both the current image and its corresponding perturbation image to the second extended training set; if not, only add the current image to the second extended training set.

[0134] In this embodiment, step 310 can be any round in multiple rounds of training. In the current training round, the current target detection model is first used to identify the relatively simple current image. If it can correctly identify it, it means that it has learned the features in the image and can learn more complex perturbation images. If the current target detection model cannot correctly identify the current image, it is only allowed to learn the relatively simple current image in this round of training, and does not learn the complex perturbation image.

[0135] In this embodiment, corresponding to the aforementioned step 308, any disturbance image and its corresponding current image have the same label category.

[0136] After obtaining the second extended training set, a corresponding loss function can be constructed according to each image sample and its label in the second extended training set to train the target detection model. This part of the content is prior art and will not be repeated here.

[0137] In another embodiment, the second extended training set includes each image in the second training set and its corresponding perturbed image. In this embodiment, corresponding to the aforementioned step 308, any perturbed image and its corresponding current image have the same label category.

[0138] In this embodiment, using the second extended training set to train the first target detection model includes:

[0139] For any current image in the second training set, determine whether the first object detection model can correctly identify the object category therein; if so, calculate the training loss based on the current image and its corresponding perturbation image; if not, calculate the training loss only based on the current image.

[0140] Through step 310, based on the idea of ​​curriculum learning, combined with the perturbed images generated in step 308 and the original images in the second training set, the first object detection model is trained from easy to difficult to obtain the second object detection model.

[0141] The method for training a target detection model provided in the embodiments of this specification is a feasible solution for small sample target detection. A tree structure diagram is constructed using prior knowledge to generate confusion categories for each new category. According to the confusion category, the generation of a perturbed image of the confusion category is guided under the adversarial sample generation mechanism. The label of the perturbed image is the label of the original image, but it is easily identified as other categories belonging to the same superclass category as the category described in the original image, which can help the target detection model better learn the boundaries of the semantic space between each category under the same superclass category.

[0142] At the same time, the embodiment of this specification provides a method for training a target detection model based on the idea of ​​curriculum learning according to the characteristics of the perturbed image, so that the model can learn the original image samples and the perturbed image samples from easy to difficult.

[0143] Finally, the method for training the target detection model provided in the embodiments of this specification is widely applicable and can be extended to different small sample target detectors, and can achieve good results.

[0144] According to an embodiment of another aspect, a device for training a target detection model is also provided. Figure 5 A schematic block diagram of an apparatus for training a target detection model according to an embodiment is shown, and the apparatus can be deployed in any device, platform or device cluster with computing and processing capabilities. Figure 5 As shown, the device 500 includes:

[0145] A first acquisition unit 502 is configured to acquire a first object detection model trained based on a first training set, where each image in the first training set belongs to K object categories;

[0146] A second acquisition unit 504 is configured to acquire a second training set, wherein any image has a label category different from the K object categories;

[0147] The confusion category determination unit 506 is configured to, for any first image in the second training set, determine the super category to which its label category belongs, and determine other object categories included in the super category as the confusion category of the first image; the confusion category belongs to the K object categories;

[0148] The perturbed image determining unit 508 is configured to determine, for the first image, a perturbation value that minimizes an objective function, thereby obtaining a first perturbed image to which the perturbation value is applied; the objective function is a relative probability value between a prediction value for its label category and a prediction value for each confusion category obtained by inputting the first image to which the perturbation value is added into the first object detection model;

[0149] The model training unit 510 is configured to determine a second extended training set based on each image in the second training set and its corresponding perturbation image, and use the second extended training set to train the first object detection model to obtain a second object detection model.

[0150] According to another aspect of the embodiment, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method described in any of the above embodiments.

[0151] According to yet another embodiment, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in any one of the above embodiments is implemented.

[0152] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0153] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0154] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0155] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0156] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for training an object detection model, comprising: Obtaining a first object detection model trained based on a first training set, where each image in the first training set belongs to K object categories; Obtain a second training set, wherein any image has a label category different from the K object categories; For any first image in the second training set, determine the superclass category to which its label category belongs, and determine other object categories included in the superclass category as confusion categories of the first image; the confusion category belongs to the K object categories; For the first image, determining a disturbance value that minimizes the objective function, thereby obtaining a first disturbed image to which the disturbance value is applied; The objective function is to input the first image with the disturbance value added into the first target detection model, and obtain the relative probability value of the predicted value for its label category and the predicted value for each confusion category; Based on each image in the second training set and its corresponding perturbation image, a second extended training set is determined, and the first object detection model is trained by using the second extended training set to obtain a second object detection model.

2. The method according to claim 1, determining the superclass category to which the label category belongs, and determining other object categories included in the superclass category as the confusion category of the first image, comprising: Acquire a pre-constructed tree structure diagram, wherein the tree structure diagram includes a plurality of object categories and a super-category category to which each object category belongs; Determine, according to the tree structure diagram, the superclass category to which the label category of the first image belongs; According to the tree structure diagram, other object categories included in the super category are determined as confusion categories of the first image.

3. The method according to claim 2, wherein: The tree structure diagram is a three-layer tree structure; the root node represents all object categories, the second-layer nodes include each super-class category node, and the third-layer nodes include each object category node; any object category node is a child node of the super-class category node to which it belongs.

4. The method according to claim 1, determining the disturbance value that minimizes the objective function comprises: Get the initial disturbance value; Multiple rounds of iterations are performed, and in each round of iterations, the current disturbance value is updated in the direction in which the objective function decreases.

5. The method according to claim 4, obtaining the initial disturbance value, comprising: Sampling from a uniform distribution to obtain a first random value matrix having the same size as the first image; The first random value matrix is ​​multiplied by a preset first threshold to obtain the initial disturbance value.

6. The method according to claim 5, updating the current disturbance value, comprising: According to the objective function and the constraint domain, adjusting the disturbance value based on the projected gradient descent method; The constraint domain is determined according to the first image and the first threshold.

7. The method according to claim 1, wherein: The first target detection model includes a classifier, and the classifier includes a processing layer and an output layer; the objective function is determined by the following steps: Obtaining the predicted values ​​of each object category obtained by the processing layer, and selecting the label category and the predicted values ​​of each confusion category therefrom; The label category and the predicted values ​​of each confusion category are input into the output layer, and the function value of the objective function is determined according to the probability value output by the output layer.

8. The method according to claim 7, wherein: The output layer includes a Softmax function; determining the function value of the objective function according to the probability value output by the output layer includes: The logarithmic value of the probability value of the label category output by the Softmax function is determined as the function value of the objective function.

9. The method according to claim 1, wherein: The label of the first disturbed image is the label of the first image.

10. The method according to claim 1, determining the second extended training set comprises: For any current image in the second training set, determining whether the first object detection model can correctly identify the object category therein; If yes, both the current image and its corresponding perturbation image are added to the second extended training set; If not, then just add the current image to the second expanded training set.

11. The method according to claim 10, wherein: Any perturbed image has the same label category as its corresponding current image.

12. The method according to claim 1, wherein: A first number of images in the first training set have the same category label; a second number of images in the second training set have the same category label; the first number is greater than the second number, and the difference or ratio between the first number and the second number is greater than a preset threshold.

13. A device for training a target detection model, comprising: A first acquisition unit is configured to acquire a first target detection model trained based on a first training set, wherein each image in the first training set belongs to K object categories; A second acquisition unit is configured to acquire a second training set, wherein any image has a label category different from the K object categories; a confusion category determination unit configured to, for any first image in the second training set, determine a super category to which its label category belongs, and determine other object categories included in the super category as the confusion category of the first image; the confusion category belongs to the K object categories; A disturbance image determination unit is configured to determine, for the first image, a disturbance value that minimizes the objective function, thereby obtaining a first disturbance image to which the disturbance value is applied; The objective function is to input the first image with the disturbance value added into the first target detection model, and obtain the relative probability value of the predicted value for its label category and the predicted value for each confusion category; The model training unit is configured to determine a second extended training set based on each image in the second training set and its corresponding perturbation image, and use the second extended training set to train the first target detection model to obtain a second target detection model.

14. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 12.

15. A computing device comprising a memory and a processor, wherein: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 12 is implemented.

Images