Target detection method, model training method and device, equipment, and storage medium

Abstract

Embodiments of the present application disclose a target detection method, a training method and device of a model, equipment and a storage medium. The training method comprises: using a to-be-trained detection model to perform target detection on a training image set obtained to obtain a predicted detection result; determining an imbalance degree of positive and negative samples of different categories in the training image set; determining a loss of the predicted detection result in the different categories based on the predicted detection result and the imbalance degree of the positive and negative samples of the different categories; and updating network parameters in the to-be-trained detection model using the loss until the updated detection model meets a convergence condition.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to computer vision technology, and relate to but are not limited to a target detection method, a model training method and device, equipment, and a storage medium. BACKGROUND

[0002] The long-tail distribution of data is an important problem faced by target detection algorithms in the real world. Generally, target detection algorithms are applied in relatively balanced data distribution scenarios, while the imbalance problem caused by the long-tail distribution often greatly affects the accuracy of the target detection algorithm. Existing long-tail target detection algorithms propose sample resampling, loss reweighting and other solutions to overcome the long-tail distribution problem.

[0003] However, the existing target detection framework for long-tail distribution data is mainly based on a two-stage target detector, and there is still no related content for single-stage detection of long-tail distribution data. SUMMARY

[0004] Therefore, embodiments of the present application provide a target detection method, a model training method and device, equipment, and a storage medium.

[0005] The technical solution of the embodiments of the present application is as follows:

[0006] In a first aspect, the embodiments of the present application provide a training method of a detection model, which comprises: using a to-be-trained detection model to perform target detection on a training image set obtained to obtain a predicted detection result; determining the imbalance degree of positive and negative samples of different categories in the training image set; determining the loss of the predicted detection result in the different categories based on the predicted detection result and the imbalance degree of the positive and negative samples of the different categories; and updating network parameters in the to-be-trained detection model using the loss until the updated detection model meets a convergence condition.

[0007] In the above manner, the imbalance problem between foreground categories can be solved by using the imbalance degree of positive and negative samples of different categories.

[0008] In some embodiments, the determination of the imbalance degree of positive and negative samples of different categories in the training image set comprises: determining a cumulative gradient ratio of positive and negative samples of different categories in the training image set; and determining the imbalance degree of positive and negative samples of the different categories based on the cumulative gradient ratio of the positive and negative samples of the different categories.

[0009] In the above manner, the imbalance degree of positive and negative samples of different categories can be determined by using the cumulative gradient ratio of different categories.

[0010] In some embodiments, the determining the loss of the predicted detection result in the different categories based on the predicted detection result and the imbalance degree of the positive and negative samples in the different categories comprises: weighting the imbalance degree of the positive and negative samples in the different categories by using a first hyperparameter to obtain a weighted result of each category in the different categories; wherein the first hyperparameter is used to control the learning strength of the to-be-trained detection model on the imbalance degree of the positive and negative samples in a rare category in the different categories; and determining the loss of the predicted detection result in each category based on the predicted detection result and the weighted result of each category.

[0011] In the above manner, the imbalance degree of the positive and negative samples in different categories can be weighted by using a hyperparameter to solve the problem of sample imbalance between foreground categories.

[0012] In some embodiments, the determining the loss of the predicted detection result in the different categories based on the predicted detection result and the imbalance degree of the positive and negative samples in the different categories comprises: determining a focal loss of the predicted detection result; and adjusting the focal loss by using the imbalance degree of the positive and negative samples in the different categories to obtain the loss of the predicted detection result in the different categories.

[0013] In the above manner, the problems of sample imbalance between foreground categories and foreground-background imbalance can be solved at the same time, so that the detection model has advantages in simplicity and efficiency.

[0014] In some embodiments, the different categories at least include a rare category and a frequent category, and after the adjusting the focal loss by using the imbalance degree of the positive and negative samples in the different categories to obtain the loss of the predicted detection result in the different categories, the method further comprises: determining a weight factor based on a basic factor and the imbalance degree of the positive and negative samples in the rare category and the frequent category; wherein the basic factor is used to balance the loss contribution of the foreground region and the background region in the training image; adjusting the loss contribution of the rare category relative to the frequent category in the loss based on the weight factor to obtain an optimized loss; and the updating the network parameters in the to-be-trained detection model by using the loss comprises: updating the network parameters in the to-be-trained detection model by using the optimized loss.

[0015] In the above manner, the loss contribution of the rare category can be improved while solving the problems of sample imbalance between foreground categories and foreground-background imbalance.

[0016] In some embodiments, the updating the network parameters in the to-be-trained detection model based on the loss comprises: determining a gradient value output by the to-be-trained detection model during the updating process based on the loss; performing gradient clipping on the gradient value to obtain a clipped gradient value, in a case where the gradient value is greater than or equal to a preset threshold; and updating the network parameters in the detection model based on the clipped gradient value.

[0017] In this way, the training of the detection model can be more stable, and the fluctuation of the training result can be alleviated.

[0018] In some embodiments, the method further comprises: determining an iteration number of iterative training of the to-be-trained detection model in the pre-training phase based on a distribution of images in the training image set; pre-training the to-be-trained detection model based on the iteration number and the training image set to obtain a candidate detection model; and updating the network parameters in the to-be-trained detection model based on the loss until the updated detection model satisfies a convergence condition, comprising: updating the network parameters in the candidate detection model based on the loss until the updated detection model satisfies the convergence condition.

[0019] In this way, the appropriate pre-training iteration number can be selected according to the distribution of the training data, so that the probability of occurrence of a NaN (Not a Number) value in the training process can be reduced.

[0020] In some embodiments, the determining the iteration number of iterative training of the to-be-trained detection model in the pre-training phase based on the distribution of images in the training image set comprises: in a case where the training image set is a long-tail distribution image set, setting the iteration number of the to-be-trained detection model in the pre-training phase to a first preset value; and in a case where the training image set is a non-long-tail distribution image set, setting the iteration number to a second preset value; wherein the first preset value is greater than the second preset value.

[0021] In this way, more iteration numbers can be set for the long-tail distribution data, so that the probability of occurrence of an abnormal problem in the training process of the long-tail data can be reduced.

[0022] In some embodiments, the adopting the detection model to be trained to perform target detection on the obtained training image set to obtain a predicted detection result comprises: adopting the detection model to be trained to generate at least two prediction boxes on each region of the training image; wherein the at least two prediction boxes have different scales; determining an intersection ratio between the generated prediction boxes and the labeled boxes of the training image; and based on the intersection ratio, measuring a probability that a target in the prediction box is foreground to obtain a predicted detection result of the training image.

[0023] In this way, the sampling strategy of the detection model can be adaptively selected, and the implementation of the center-ness branch in the detection model can be changed, thereby improving the detection performance.

[0024] In some embodiments, the method further comprises: pre-processing each training image in the training image set to obtain a pre-processed training image; and performing feature extraction on the pre-processed training image to obtain a feature map; and the adopting the detection model to be trained to generate at least two prediction boxes on each region of the training image comprises: adopting the detection model to be trained to generate at least two prediction boxes on each pixel of the feature map; wherein the pixels of the feature map have a corresponding relationship with the regions in the training image.

[0025] In this way, a complete model training process can be implemented.

[0026] In a second aspect, the embodiments of the present application provide a target detection method, which comprises: obtaining a to-be-detected image; pre-processing the to-be-detected image to obtain a pre-processed to-be-detected image; performing feature extraction on the pre-processed to-be-detected image to obtain a feature map of the to-be-detected image; inputting the feature map of the to-be-detected image into a detection model to obtain a prediction result; wherein the detection model is obtained based on the training method described above; and post-processing the prediction result to obtain a detection result of the to-be-detected image.

[0027] In this way, the detection model obtained based on the training method described above can be used to process the to-be-detected image, thereby improving the detection rate.

[0028] In some embodiments, the prediction result comprises a plurality of prediction boxes and a target category corresponding to each prediction box; and the post-processing the prediction result to obtain the detection result of the to-be-detected image comprises: filtering the prediction boxes that have overlaps in the plurality of prediction boxes to obtain filtered prediction boxes and a target category corresponding to each filtered prediction box.

[0029] In this way, the plurality of prediction boxes obtained by the detection model can be de-duplicated to obtain a more accurate detection result.

[0030] In a third aspect, an embodiment of the present application provides a training device of a detection model, the device comprising: a first prediction unit configured to perform target detection on a training image set obtained by using a detection model to be trained, to obtain a predicted detection result; a degree determination unit configured to determine an imbalance degree of positive and negative samples of different categories in the training image set; a loss determination unit configured to determine a loss of the predicted detection result in the different categories based on the predicted detection result and the imbalance degree of the positive and negative samples of the different categories; and a parameter updating unit configured to update network parameters in the detection model to be trained by using the loss until the updated detection model meets a convergence condition.

[0031] In a fourth aspect, an embodiment of the present application provides a target detection device, the device comprising: an acquisition unit configured to acquire a to-be-detected image; a preprocessing unit configured to perform preprocessing on the to-be-detected image to obtain a preprocessed to-be-detected image; a feature extraction unit configured to perform feature extraction on the preprocessed to-be-detected image to obtain a feature map of the to-be-detected image; a second prediction unit configured to input the feature map of the to-be-detected image into a detection model to obtain a prediction result, wherein the detection model is trained based on the training method described above; and a post-processing unit configured to perform post-processing on the prediction result to obtain a detection result of the to-be-detected image.

[0032] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program capable of running on the processor, and the processor implements the steps in the training method described above or the steps in the detection method described above when executing the program.

[0033] In a sixth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps in the training method described above or the steps in the detection method described above.

[0034] The embodiments of the present application provide a target detection method, a training method and device of a model, an electronic device and a computer readable storage medium. The training image set obtained is detected by using the detection model to be trained, to obtain a predicted detection result. The imbalance degree of positive and negative samples of different categories in the training image set is determined. The loss of the predicted detection result in the different categories is determined based on the predicted detection result and the imbalance degree of the positive and negative samples of the different categories. The network parameters in the detection model to be trained are updated by using the loss until the updated detection model meets a convergence condition. In this way, the imbalance of samples between foreground categories can be solved by using the imbalance degree of positive and negative samples of different categories. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 This is a schematic diagram illustrating the implementation process of the training method for the detection model in the embodiments of this application. Figure 1 ;

[0036] Figure 2 This is a schematic diagram illustrating the implementation process of the training method for the detection model in the embodiments of this application. Figure 2 ;

[0037] Figure 3 This is a schematic diagram illustrating the implementation process of the training method for the detection model in an embodiment of this application. Figure 3 ;

[0038] Figure 4 This is a schematic diagram illustrating the implementation process of the target detection method in an embodiment of this application;

[0039] Figure 5A This is a structural block diagram of a single-stage long-tail target detection scheme according to an embodiment of this application;

[0040] Figure 5B This is a comparison diagram of the sample distribution of the single-stage detector and the two-stage detector in the embodiments of this application;

[0041] Figure 5C This is a schematic diagram illustrating the proportion of samples with different data distributions in an embodiment of this application.

[0042] Figure 5D This is a schematic diagram of the equilibrium focus loss function in an embodiment of this application. Figure 1 ;

[0043] Figure 5E This is a schematic diagram of the equilibrium focus loss function in an embodiment of this application. Figure 2 ;

[0044] Figure 6 This is a schematic diagram of the composition of the training device for the detection model in an embodiment of this application;

[0045] Figure 7 This is a schematic diagram of the composition and structure of the target detection device according to an embodiment of this application;

[0046] Figure 8 This is a schematic diagram of a hardware entity of an electronic device according to an embodiment of this application. Detailed Implementation

[0047] The technical solutions of this application will be further described in detail below with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0048] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" can be the same subset or different subsets as each other and can be combined with each other as long as there is no conflict.

[0049] In the following description, the suffixes such as "module", "component", or "unit" used to represent elements are used only for the convenience of description of the present application, and have no specific meaning in themselves. Therefore, "module", "component", or "unit" can be used interchangeably.

[0050] It should be noted that the terms "first", "second", "third" involved in the embodiments of the present application are only to distinguish similar objects, and do not represent a specific order of the objects. Understandably, "first", "second", "third" can be interchanged in a specific order or sequence as long as it is allowed, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0051] Single-stage target detectors are simple, fast, and easy to deploy. However, the performance of current single-stage target detectors is often inferior to that of two-stage detectors, and single-stage target detection models mainly face two difficulties in the case of long-tail distribution data. First, the long-tail distribution of data brings the problem of sample imbalance between foreground classes, which is an inherent problem in long-tail distribution scenarios. Second, single-stage target detection algorithms are based on dense sample candidate sets, which introduce foreground-background imbalance problems in the training process of the model.

[0052] These two imbalance problems together hinder the performance of single-stage target detection models in long-tail scenarios. Existing solutions generally only focus on the sample imbalance problem between foreground classes, such as EQLv2; or only focus on the foreground-background imbalance problem, such as focal loss (Focal Loss). The prior art lacks the ability to solve both imbalance problems at the same time, so the prior art cannot well handle the single-stage long-tail target detection task.

[0053] Based on this, the embodiments of the present application provide a training method of a detection model, which is applied to an electronic device. The functions implemented by the method can be realized by calling program codes by a processor in the electronic device. Of course, the program codes can be saved in a storage medium of the electronic device. Figure 1 The implementation process of the training method of the detection model of the embodiments of the present application is shown in the following figure Figure 1 As shown in the figure, the method comprises the following steps: Figure 1

[0054] Step S101, using a detection model to be trained, performing target detection on a set of training images obtained to obtain a predicted detection result; ​

[0055] Here, the electronic device can be various types of devices with information processing capabilities, such as a navigator, a smart phone, a tablet computer, a wearable device, a laptop computer, an all-in-one computer and a desktop computer, a server cluster, and the like.

[0056] In an embodiment of the present application, the training image set can be a commonly used data set in the field of computer vision, such as a COCO data set, an LVIS data set, and the like. Of course, the training image set can also be a collection of images in a real scene, such as images taken in the field of autonomous driving. Moreover, the data distribution of the images in the training image set can be a long-tail distribution.

[0057] Here, the detection model to be trained can be a single-stage detection model or a two-stage detection model, and the embodiments of the present application do not limit this. In the case where the detection model to be trained is a single-stage detection model, the single-stage detection model can include a RetinaNet (Focal Loss for Dense Object Detection) single-stage detector, an ATSS (Adaptive Training Sample Selection) single-stage detector, a YOLO (You Only Look Once) single-stage detector, and the like. The predicted detection result can include detected targets and the target category to which each target belongs.

[0058] Step S102, determine the imbalance degree of positive and negative samples of different categories in the training image set;

[0059] For example, there are three categories of target objects that need to be detected: the first category of target objects (such as cats), the second category of target objects (such as pandas), and the third category of target objects (such as snow leopards). There are 225 images including the first category of target objects, 115 images including the second category of target objects, and 5 images including the third category of target objects in the training image set. Different categories in the training image set refer to the three categories, and the first category is a frequent category, the second category is an ordinary category, and the third category is a rare category. The training image set can be regarded as a long-tail distribution data set.

[0060] Further, the imbalance degree of the positive and negative samples of different categories in the training image set refers to that when the training image set is a long-tail distribution dataset, not only each category has a positive and negative sample distribution imbalance, but also the rare category faces a more serious positive and negative sample imbalance problem than the frequent category. That is, in the training image set, the imbalance degree of the positive and negative samples of the third category is greater than that of the second category, and the imbalance degree of the positive and negative samples of the second category is greater than that of the first category.

[0061] Here, the positive and negative sample imbalance refers to that the number of negative samples (background samples) is much greater than the number of positive samples (foreground samples).

[0062] Step S103, determining a loss of the predicted detection result in the different categories based on the predicted detection result and the imbalance degree of the positive and negative samples of different categories;

[0063] Here, the loss of the detection result of the training sample set in different categories can be determined according to the predicted detection result and the imbalance degree of the positive and negative samples of different categories. For example, the loss of the detection result of the training sample set in the first category, the loss of the detection result of the training sample set in the second category, and the loss of the detection result of the training sample set in the third category. That is, each of the different categories corresponds to a loss value.

[0064] In the embodiment of the present application, the loss is determined by the imbalance degree of the positive and negative samples of different categories, so that the detection model improves the learning intensity of the rare category relative to the frequent category, thereby solving the sample imbalance problem between foreground categories and improving the detection effect of the detection model on the rare category.

[0065] Step S104, updating the network parameters in the detection model to be trained using the loss until the updated detection model meets a convergence condition.

[0066] Here, the updated detection model meeting the convergence condition can be realized in the following three ways: the first is that the loss output by the detection model is less than a certain preset value; the second is that the change of the weight value between two iterations is less than a certain preset value; and the third is that the number of iterations reaches a preset number.

[0067] In some embodiments, the step S102 of determining the imbalance degree of the positive and negative samples of different categories in the training image set comprises:

[0068] Step S1021, determining a cumulative gradient ratio of the positive and negative samples of different categories in the training image set;

[0069] In step S1022, the imbalance degree of the positive and negative samples of the different categories is determined based on the accumulated gradient ratio of the positive and negative samples of the different categories.

[0070] Here, the accumulated gradient ratio of the positive and negative samples of each category in the training process can be determined, and the accumulated gradient ratio of the positive and negative samples of each category is used to measure the imbalance degree of the positive and negative samples of the corresponding category.

[0071] In some embodiments, the method further comprises:

[0072] In step S11, the number of iterations for iteratively training the detection model to be trained in the pre-training stage is determined based on the distribution of the images in the training image set.

[0073] In step S12, the detection model to be trained is pre-trained based on the number of iterations and the training image set to obtain a candidate detection model.

[0074] Correspondingly, the step S104 of updating the network parameters in the detection model to be trained using the loss until the updated detection model satisfies the convergence condition comprises: updating the network parameters in the candidate detection model using the loss until the updated detection model satisfies the convergence condition.

[0075] Here, the pre-training stage, i.e., the warm up stage, is to train with a very small learning rate at the beginning of training, so that the network is familiar with the data. As the training proceeds, the learning rate gradually increases, and when it reaches a certain level, the training is performed with the set initial learning rate, and then the learning rate gradually decreases. That is, after enabling warm up, the number of iterations of warm up needs to be set. In the case where the current number of iterations is less than the set number of iterations, the learning rate is equal to the current number of iterations divided by the set number of iterations multiplied by the base learning rate. Since the value of the current number of iterations divided by the set number of iterations is a number less than 1, the learning rate is an increasing process in the whole warm up process. When the warm up ends, the learning rate is trained with the base learning rate.

[0076] Therefore, in the embodiments of the present application, the number of iterations of the detection model in the pre-training stage can be determined according to the distribution of the images in the training image set (for example, whether the data in the training image set is a long-tail distribution), and then pre-training is performed, thereby reducing the probability of a series of abnormal problems (for example, the probability of NAN data) occurring in the training process. If the training data is a long-tail distribution, the preset number of iterations can be set to a larger value, so that the warm up time is longer.

[0077] For example, the original network 1000 iterations to learn rate is 0.01, the application is set to 6000 times to reach 0.01 learning rate. Thus, the time of warm up will increase, and then will reduce the probability of NAN data in the training process.

[0078] In some embodiments, the step S11, based on the distribution of images in the training image set, determines the number of iterations of the pre-training phase for the to-be-trained detection model.

[0079] Step S11a, in the case of the training image set being a long-tail distribution image set, the number of iterations of the to-be-trained detection model in the pre-training phase is set to a first preset value;

[0080] Step S11b, in the case of the training image set being a non-long-tail distribution image set, the number of iterations is set to a second preset value; wherein the first preset value is greater than the second preset value.

[0081] Based on the foregoing embodiments, the embodiments of the application further provide a detection model training method, which is applied to an electronic device, and the method comprises:

[0082] Step S111, using a to-be-trained detection model, performing target detection on the obtained training image set to obtain a predicted detection result;

[0083] Step S112, determining the cumulative gradient ratio of positive and negative samples of different categories in the training image set;

[0084] Step S113, based on the cumulative gradient ratio of the positive and negative samples of different categories, determining the imbalance degree of the positive and negative samples of different categories;

[0085] Step S114, using a first hyperparameter to weight the imbalance degree of the positive and negative samples of different categories respectively to obtain a weighted result of each category in the different categories; wherein the first hyperparameter is used to control the learning intensity of the to-be-trained detection model on the imbalance degree of the positive and negative samples of rare categories in different categories;

[0086] Here, the cumulative gradient ratio is related to the balance degree of the training process. The cumulative gradient ratio is close to 1 for the balanced categories in the training process; and the cumulative gradient ratio is close to 0 for the unbalanced categories in the training process. Then, the unbalanced categories in the training process need to be weighted to control the maximum intensity of learning the imbalance problem of the positive and negative samples of the unbalanced categories (such as rare categories).

[0087] Step S115, based on the predicted detection result and the weighted result of each category, determining the loss of the predicted detection result in each category.

[0088] Step S116, updating the network parameters in the to-be-trained detection model using the loss until the updated detection model meets a convergence condition.

[0089] Based on the foregoing embodiments, the embodiments of the present application further provide a detection model training method. The method is applied to an electronic device and includes:

[0090] Step S121, performing target detection on the obtained training image set using the to-be-trained detection model to obtain a predicted detection result.

[0091] Step S122, determining the imbalance degree of positive and negative samples of different categories in the training image set.

[0092] Step S123, determining the focal loss of the predicted detection result.

[0093] Here, the focal loss of the predicted detection result can be determined using the focal loss function in formula (1) as follows:

[0094] ................................ (1).

[0095] wherein, is the focal loss, is used to balance the relationship between positive samples and negative samples in each category, is the probability of determining the target in the predicted bounding box as foreground, is a focal parameter, which is used to reduce the weight of easy-to-classify samples so that the model focuses more on difficult-to-classify samples during training.

[0096] Step S124, adjusting the focal loss using the imbalance degree of positive and negative samples of different categories to obtain the loss of the predicted detection result in the different categories.

[0097] Here, since the loss in the different categories is obtained by adjusting the focal loss using the imbalance degree of positive and negative samples of different categories, the detection model during training can not only balance the loss contribution of foreground samples and background samples, but also improve the learning intensity of rare categories relative to frequent categories, thereby solving the foreground-background imbalance problem and the sample imbalance problem among foreground categories.

[0098] Step S125, updating the network parameters in the to-be-trained detection model using the loss until the updated detection model meets a convergence condition.

[0099] In some embodiments, a loss function including a focus factor is introduced based on the above-mentioned focus loss. This focus factor is used to adjust the learning intensity of different categories of samples according to the degree of imbalance between positive and negative samples of different categories in the training image set. Specifically, for rare categories, the focus factor emphasizes increasing the learning intensity of positive samples while reducing the learning attention to negative samples. For frequent categories, the focus factor maintains a similar performance to the traditional focus loss. In this way, the two major imbalance problems can be reduced to the problem of inconsistent imbalance between positive and negative samples of different categories, and the proposed loss function including the focus factor can simultaneously solve the two major problems of sample imbalance between foreground categories and foreground-background imbalance. The new loss function can be obtained by the following formula (2):

[0100] ................................(2;

[0101] Among them, the It is a basic factor used to balance the loss contributions of foreground and background regions in the training image; the The first hyperparameter controls the strength with which the detection model to be trained learns the imbalance between positive and negative samples of rare classes in different categories; For category indexes of different categories, the It is the first in the training process The cumulative gradient ratio of positive and negative samples in each class; The focusing factor; This is a new loss function proposed in the embodiments of this application.

[0102] Based on the foregoing embodiments, this application further provides a method for training a detection model, which is applied to an electronic device. Figure 2 This is a schematic diagram illustrating the implementation process of the training method for the detection model in an embodiment of this application. Figure 2 ,like Figure 3 As shown, the method includes:

[0103] Step S201: Using the detection model to be trained, perform target detection on the acquired training image set to obtain the predicted detection results;

[0104] Step S202: Determine the degree of imbalance between positive and negative samples of different categories in the training image set;

[0105] Step S203: Determine the focus loss of the predicted detection result;

[0106] Step S204, adjusting the focal loss by using the imbalance degree of the positive and negative samples of the different categories to obtain the loss of the predicted detection result in the different categories;

[0107] Step S205, determining a weight factor based on a basic factor and the imbalance degree of the positive and negative samples of the rare category and the frequent category in the different categories; wherein the basic factor is used to balance the loss contribution of the foreground region and the background region in the training image.

[0108] Here, the weight factor is mainly used to improve the loss contribution of the rare category relative to the frequent category.

[0109] Step S206, adjusting the loss contribution of the rare category relative to the frequent category in the loss based on the weight factor to obtain an optimized loss.

[0110] Here, the different categories at least include the rare category and the frequent category, and thus the loss contribution of the rare category relative to the frequent category in the loss obtained in the above step S204 can be adjusted based on the weight factor.

[0111] Step S207, updating the network parameters in the detection model to be trained by using the optimized loss until the updated detection model meets a convergence condition.

[0112] In some embodiments, the new loss function in the above formula (2) is optimized to obtain an optimized new loss function. The optimized new loss function can be obtained by the following formula (3):

[0113] ..................(3);

[0114] Wherein, the is the total number of categories; the is the weight factor.

[0115] The reason for optimizing the new loss function in formula (2) is that although the focus factor can dynamically adjust the learning intensity of each class sample according to the positive and negative sample imbalance degree of the class, the focus factor alone cannot make the detection model achieve optimal performance. Because in the scene of classifying multiple classes together, the larger the focus factor, the smaller the loss contribution in the training process. The rare class needs to use a large focus factor to learn its extreme positive and negative sample imbalance problem, but it will lose gradient contribution because of the large focus factor. And when the rare class difficult sample and the frequent class difficult sample learn together, even if each class has a different focus factor, the loss contribution of the two is basically the same. Therefore, the embodiment of the application further proposes a weight factor on the basis of the loss function including the focus factor, so that the rare class difficult sample has more loss contribution than the frequent class difficult sample.

[0116] Based on the foregoing embodiments, the embodiments of the application further provide a detection model training method, which is applied to an electronic device, and the method comprises:

[0117] Step S211, using a detection model to be trained, performing target detection on the obtained training image set to obtain a predicted detection result;

[0118] Step S212, determining the imbalance degree of positive and negative samples of different classes in the training image set;

[0119] Step S213, based on the predicted detection result and the imbalance degree of positive and negative samples of different classes, determining the loss of the predicted detection result in the different classes;

[0120] Step S214, based on the loss, determining the gradient value output by the detection model to be trained in the updating process;

[0121] Step S215, in the case where the gradient value is greater than or equal to a preset threshold, performing gradient clipping on the gradient value to obtain a clipped gradient value;

[0122] Here, because there is gradient propagation during training, excessive gradient leads to unstable training results, so it needs to be processed. When the gradient is very large during training, the gradient is directly cut off, for example, when the training gradient is 100, it is cut to 35. Especially in the case of long-tail distribution of data, the gradient is relatively unstable, so the gradient clipping strategy is added in the training stage in the embodiments of the application. In this way, the single-stage detector training is more stable, and the training result fluctuation is alleviated.

[0123] Step S216, based on the clipped gradient value, updating the network parameters in the detection model until the updated detection model meets the convergence condition.

[0124] Based on the foregoing embodiments, the embodiments of the present application further provide a training method of a detection model, the method being applied to an electronic device, Figure 3 The implementation process of the training method of the detection model of the embodiments of the present application is shown in the figure Figure 4 As shown in the figure, Figure 4 The method comprises the following steps:

[0125] Step S301, using a detection model to be trained, at least two prediction boxes are generated on each region of the obtained training image; wherein the at least two prediction boxes have different scales;

[0126] For example, the detection model to be trained comprises an ATSS detector, the existing ATSS detector is improved on the basis of a RetinaNet network and an FCOS (Fully Convolutional One-Stage Object Detection) algorithm, the ATSS detector only lays 1 anchor box on each pixel point of a feature map, and 9 anchor boxes are selected as positive samples at each layer. In the embodiments of the present application, the existing ATSS detector is improved, at least 2 anchor boxes are laid on each pixel point of the feature map, and the at least 2 anchor boxes have different scales. For example, two anchor boxes are laid at each point, one anchor box has a scale of 6, and the other anchor box has a scale of 8, and then the first 18 samples at each layer are taken as positive samples, so that the performance of the ATSS detector can be improved.

[0127] Step S302, determining the intersection ratio between the generated prediction box and the labeled box of the training image;

[0128] Here, the labeled box of the training image can be a GT box of the training image, i.e. a ground truth box.

[0129] Step S303, based on the intersection ratio, the probability that the target in the prediction box is foreground is measured, and the detection result predicted by the training image is obtained;

[0130] For example, the detection model to be trained comprises an ATSS detector, the existing ATSS detector comprises three branches, i.e. a classification branch, a positioning branch and a center branch, the center branch (center-ness) is used to determine the probability that the target in each prediction box is foreground or background, and further distinguishes foreground and background, and the existing implementation manner is to score the position. In the embodiments of the present application, the existing ATSS detector is improved, and the implementation manner of the center branch is changed to the mode of using the intersection ratio IOU (Intersection over Union).

[0131] Step S304, determining an imbalance degree of positive and negative samples of different categories in the training image set;

[0132] Step S305, determining a loss of the predicted detection result in the different categories based on the predicted detection result and the imbalance degree of the positive and negative samples of the different categories;

[0133] Step S306, updating network parameters in the detection model to be trained by using the loss until the updated detection model meets a convergence condition.

[0134] Based on the foregoing embodiments, the embodiments of the present application further provide a detection model training method, which is applied to an electronic device and includes the following steps:

[0135] Step S311, pre-processing each training image in the obtained training image set to obtain a pre-processed training image;

[0136] Here, the pre-processing of the training image includes but is not limited to scaling of image pixels, image flipping, and conversion of image data forms.

[0137] Step S312, performing feature extraction on the pre-processed training image to obtain a feature map;

[0138] Here, the training image needs to be subjected to feature extraction, and then the extracted feature map is input into the detection model for processing. It should be noted that the method for performing feature extraction in the embodiments of the present application is not limited. For example, the method of combining backbone and FPN (Feature Pyramid Networks) can be used for feature extraction.

[0139] Step S313, generating at least two prediction boxes on each pixel of the feature map by using a detection model to be trained; wherein the at least two prediction boxes have different scales; the pixels of the feature map have a corresponding relationship with regions in the training image;

[0140] Here, at least two prediction boxes can be generated on each pixel of the feature map. Since the size of the feature map is much smaller than the size of the original training image, one pixel of the feature map corresponds to a region in the original training image.

[0141] Step S314, determining a intersection-over-union ratio between the generated prediction boxes and labeled boxes of the training image;

[0142] Step S315, measuring a probability that a target in the prediction box is a foreground based on the intersection-over-union ratio to obtain a predicted detection result of the training image.

[0143] Step S316, determining the imbalance degree of positive and negative samples of different categories in the training image set;

[0144] Step S317, determining the loss of the predicted detection result in the different categories based on the predicted detection result and the imbalance degree of positive and negative samples of different categories;

[0145] Step S318, updating the network parameters in the detection model to be trained using the loss until the updated detection model meets the convergence condition.

[0146] Based on the foregoing embodiments, the embodiments of the present application provide a target detection method, which is applied to an electronic device. The functions implemented by the method can be realized by calling program codes by a processor in the electronic device. Of course, the program codes can be saved in a storage medium of the electronic device. Figure 5A For the implementation process of the target detection method of the embodiments of the present application, as shown in Figure 5A The method comprises the following steps:

[0147] Step S401, obtaining a to-be-detected image;

[0148] Here, the to-be-detected image can be a test image in a commonly used data set in the field of computer vision, such as a COCO data set, an LVIS data set, etc. Of course, the to-be-detected image can also be an image in a real scene, such as an image taken in the field of automatic driving.

[0149] Step S402, pre-processing the to-be-detected image to obtain a pre-processed to-be-detected image;

[0150] Here, the pre-processing of the to-be-detected image includes not only scaling of image pixels, flipping of images, conversion of image data forms, but also adjustment of image angles, filtering of images, etc.

[0151] Step S403, performing feature extraction on the pre-processed to-be-detected image to obtain a feature map of the to-be-detected image;

[0152] Step S404, inputting the feature map of the to-be-detected image into a detection model to obtain a prediction result; wherein the detection model is trained based on the training method described above;

[0153] In the embodiments of the present application, the detection model trained by the training method can be used to process the to-be-detected image to obtain a plurality of prediction boxes and a category of each prediction box. For example, after the to-be-detected image is processed by using the trained detection model, 8 prediction boxes are obtained, 2 of which are background and 6 of which are foreground, and in the 6 foreground prediction boxes, 4 of them include the same kitten and 2 of them include the same dog.

[0154] In step S405, the prediction result is post-processed to obtain the detection result of the to-be-detected image.

[0155] For example, after the to-be-detected image is processed by using the trained detection model, 8 prediction boxes are obtained, 2 of which are background and 6 of which are foreground, and in the 6 foreground prediction boxes, 4 of them include the same kitten and 2 of them include the same dog. Then, the detection result is processed to filter the background boxes and the overlapping foreground boxes, and only one foreground prediction box including the kitten and one foreground prediction box including the dog are reserved, which are taken as the final detection result of the to-be-detected image.

[0156] In some embodiments, the prediction result includes a plurality of prediction boxes and a target category corresponding to each prediction box; and the step S405 of post-processing the prediction result to obtain the detection result of the to-be-detected image includes: filtering the prediction boxes that exist in the plurality of prediction boxes to obtain filtered prediction boxes and a target category corresponding to each filtered prediction box.

[0157] For example, the NMS (Non-Maximum Suppression) method can be used to filter the prediction boxes that exist in the plurality of prediction boxes to obtain the detection result of the to-be-detected image.

[0158] Based on the foregoing embodiments, the embodiments of the present application further provide a target detection method, which is applied to a single-stage long-tail distribution scene.

[0159] The target detection method mainly includes the following contents:

[0160] (1) For the high-performance baseline problem, the embodiments of the present application propose an enhanced baseline based on the existing single-stage detectors, which can achieve a performance comparable to that of a two-stage detector, facilitating subsequent research.

[0161] (2) In view of the two imbalance problems of imbalance between samples of foreground categories and imbalance between foreground and background samples, an embodiment of the present application proposes a new loss function, namely EFL (Equalized Focal Loss), which contains two key adjustment factors, namely focusing factor and weighting factor, and can solve the above two imbalance problems.

[0162] (3) In combination with the enhanced baseline and the EFL, an embodiment of the present application provides a complete single-stage long-tail target detection scheme, which has superior performance in a long-tail scene.

[0163] Figure 5B The structural block diagram of the single-stage long-tail target detection scheme of the embodiment of the present application is shown in FIG. 1, which mainly includes four parts, which are described in detail as follows: Figure 5B

[0164] 1) The first part 51 mainly includes three modules, namely an image reading module 511, a pre-processing module 512 and a feature extraction module 513. The image reading module 511 is mainly used to read original training images or images to be detected; the pre-processing module 512 is mainly used to pre-process the original training images or images to be detected, and the pre-processing includes scaling operation on image pixels, flipping operation on images, conversion operation on data form of images, etc.; the feature extraction module 513 is mainly used to extract features of the pre-processed images, for example, the feature extraction module can include backbone and FPN.

[0165] Here, the image reading module 511, the pre-processing module 512 and the feature extraction module 513 can be multi-process parallel. Moreover, the images read by the image reading module 511 are long-tail distributed image data.

[0166] 2) The second part 52: The input of the second part is the feature map output by the first part, and the second part mainly includes an ATSS single-stage detector 521, and a stability setting 522 and an enhancement setting 523 for the ATSS single-stage detector. The ATSS single-stage detector 521, the stability setting 522 and the enhancement setting 523 together constitute an enhanced baseline.

[0167] In the scene where the data is long-tail distributed, the performance of most single-stage detectors is not as good as that of two-stage detectors, so an embodiment of the present application improves this problem, and the corresponding improvement method includes:

[0168] ​The embodiment of the present application adds a stability setting 522 for stabilizing training on the basis of the existing ATSS single-stage detector 521. The stability setting 522 includes adding a gradient clipping strategy with a maximum regularization value of a first value. For example, the first value is 35, and when the gradient of 100 is trained, it is clipped to 35. The stability setting 522 also includes extending the pre-training stage (warm-up stage) from a first number of iterations to a second number of iterations, where the first number is less than the second number. For example, the number of iterations of the pre-training stage is extended from 1000 to 6000 iterations. The stability setting 522 makes the training process of the ATSS single-stage detector 521 more stable, and also alleviates the fluctuation of the training result, reduces the occurrence of NaN values in the training process, and a series of problems.

[0169] The embodiment of the present application also adds an enhancement setting 523 to the existing ATSS single-stage detector 521. The enhancement setting 523 includes replacing the center branch (Centerness branch) of the ATSS single-stage detector 521 with an IoU branch, and adjusting the scale of the predicted box (anchor box) of the ATSS single-stage detector 521 from 8 to 6 and 8. Furthermore, the first 18 samples are used as positive samples for sampling training with such a sampling strategy.

[0170] In this way, the ATSS single-stage detector 521 combines the stability setting 522 and the enhancement setting 523 to form an enhanced baseline. The enhanced baseline proposed by the embodiment of the present application achieves a performance comparable to that of a two-stage detector.

[0171] 3) The third part 53: the embodiment of the present application provides a new loss function, namely the balanced focal loss EFL.

[0172] Figure 5C For the sample distribution comparison chart of the single-stage detector and the two-stage detector of the embodiment of the present application, as shown in Figure 5C The two-stage detector generally does not have the foreground-background imbalance problem because it has an RPN (Region Proposal Network) that filters out most of the background samples before the final classification, so the two-stage classifier acts on a relatively balanced distribution of foreground and background. Since the single-stage detector directly performs dense prediction on the feature map, there is a relatively extreme foreground-background imbalance problem, that is, the number of negative samples is much larger than that of positive samples.

[0173] In combination with the foreground class sample imbalance problem introduced in the case of long-tail distribution, the two imbalance problems jointly hinder the performance of the single-stage long-tail detector.

[0174] Because the above two imbalance problems interact with each other in the single-stage detector, they are coupled, so an algorithm is needed to solve the two problems at the same time. Existing solutions often only have the ability to solve the foreground-background imbalance problem or only have the ability to solve the sample imbalance problem between foreground classes.

[0175] Figure 5C For the number ratio of samples with different data distributions in the embodiments of the present application, as shown in FIG. 5, the horizontal axis is the class index, the vertical axis is the ratio, curve 501 is a long-tail distribution data set, and curve 502 is a uniform distribution data set. From the perspective of the horizontal axis, curve 501 is declining, that is, the classes in the front are more than the classes in the back, indicating that the sample classes in the long-tail distribution data set are imbalanced. From the vertical axis, it can be seen that the foreground and background are imbalanced, and there are too many negative samples and too few positive samples. Figure 5D

[0176] And, from the above analysis, it can be analyzed that the distribution of positive and negative samples of each class under the long-tail distribution, and through the analysis, it is found that although the positive and negative samples of each class in the long-tail distribution are imbalanced, the rare classes obviously face more serious positive and negative sample imbalance problems than the frequent classes. Therefore, in the embodiments of the present application, the two imbalance problems of the single-stage detection model under the long-tail distribution are ultimately attributed to the problem that the positive and negative sample imbalance degrees of different classes are inconsistent. Figure 1 The focal loss equally handles the positive and negative sample imbalance problems of all classes, and cannot handle the above problem that the positive and negative sample imbalance degrees of different classes are inconsistent.

[0177] Based on this analysis, the embodiments of the present application propose an equal focal loss EFL based on the above focal loss, which can be obtained from the following formula (4):

[0178]

[0179] ...................................(4);

[0180] Among them, is the first important factor proposed in the embodiments of the present application, that is, the focusing factor, which can dynamically adjust the learning intensity of each class sample according to the positive and negative sample imbalance degree of the class. represents the loss function in the i-th class.

[0181] ​​​For rare classes, the balanced focus loss emphasizes increasing the learning intensity for positive samples while reducing the learning focus on negative samples. For frequent classes, the balanced focus loss maintains similar performance to the focus loss described above.

[0182] Furthermore, in this embodiment, the focusing factor is divided into two parts: one part is a category-independent basic factor. The other part is category-related factors. The aforementioned The basic behavior of the equalization focus loss used to control the balance of positive and negative samples in handling the imbalance problem; Following the gradient weighting mechanism proposed in EQLv2, a large value is assigned to the rare class to amplify the attention given to the imbalance between positive and negative samples of the rare class, while a small value is assigned to the frequent class to maintain the basic behavior of the loss function. The focusing factor can be obtained from the following formula (5):

[0183] .......................................(5)

[0184] in, It is a hyperparameter that controls the maximum strength of the balanced focus loss in learning the imbalance problem of positive and negative samples in rare classes. It is the first step in the training process The cumulative gradient ratio of positive and negative samples in each class.

[0185] Figure 5D This is a schematic diagram of the equilibrium focus loss function in an embodiment of this application. Figure 5D ,like Figure 5D As shown, the horizontal axis is The The features are those that enter the classifier, where The vertical axis represents the loss. Figure 5E Different curves in the graph represent different sample categories. From... Figure 2 As can be seen, focusing factors alone are insufficient to achieve optimal performance in a single-stage model. There are two main reasons for this: one is in scenarios involving the co-classification of multiple categories... A larger value results in a smaller contribution to the loss during training; rare classes require a larger value. To learn about its extreme problem of imbalanced positive and negative samples, but it will also be affected by large... And the loss gradient contribution. Another reason is that when hard samples from rare classes are learned together with hard samples from frequent classes, even if each class has different gradients... Finally, the loss contributions of both are basically the same. In fact, because the number of rare difficult samples is small, when classifying, it is more desirable for them to have more loss contribution than frequent difficult samples.

[0186] Based on the above two reasons, the embodiment of the present application adds a weighting factor to the balanced focal loss, which can increase the loss contribution of the rare class and maintain the loss contribution of the original frequent class. The balanced focal loss after adding the weighting factor (the final EFL) can be obtained by the following formula (6):

[0187] .......................(6);

[0188] Wherein, C is the total number of classes, is the weighting factor.

[0189] Figure 5E The balanced focal loss function of the embodiment of the present application Figure 5D As Figure 5E shown, the horizontal axis is , and the vertical axis is the loss. Figure 5D And Figure 5E In both, the rare class corresponding to is given a larger value to focus on the foreground-background imbalance problem, and the common class corresponding to is given a smaller value. Figure 5D In, when the value of the horizontal axis is gradually negative, the loss function of all classes will converge to the same value, which is not expected, because The smaller the value, the more difficult the sample is to learn. Figure 5E In, when the horizontal and vertical are gradually negative, the convergence degree of each class sample is different, the rarer the class, the greater the loss, the more common the class, the smaller the loss, and the relative relationship between the loss corresponding to each class is also different. Therefore, compared with Figure 6 and Figure 6 It can be seen that when the weighting factor is added to the balanced focal loss, the attention of the rare class is higher than that of the common class, and the loss contribution is also more.

[0190] It can be seen that the balanced focal loss in formula (3) simultaneously solves the sample imbalance problem among foreground classes in the single-stage long-tail target detection scene and the foreground-background imbalance problem. Furthermore, in combination with the enhanced baseline, the single-stage long-tail target detection scheme proposed in the embodiments of the present application surpasses all existing long-tail related algorithms. In addition, in the scene distribution of balanced foreground class samples, the balanced focal loss in the embodiments of the present application is equivalent to the focal loss, and this feature enables the balanced focal loss proposed in the embodiments of the present application to be combined with any sampler (for example, a random sampler, a class-balanced sampler, a class-weighted sampler, etc.) and work well on any data distribution.

[0191] 4) The fourth part 54: if the first part input is a to-be-detected image, the second part output is a prediction box corresponding to the to-be-detected image, the prediction box includes a foreground box and a background box, and correspondingly, the output of the second part also includes the class of the foreground in each foreground box. However, there are many redundant prediction boxes (for example, prediction boxes with high overlap) in the output prediction box, so the fourth part needs to further process the output of the second part. Furthermore, the fourth part mainly includes a post-processing module 541 and an output module 542, the post-processing module 541 is used for post-processing the output of the second part, for example, performing an NMS operation on the output of the second part; and the output module 542 is used for outputting the detection result of the to-be-detected image.

[0192] Here, the first part 51, the second part 52 and the third part 53 together constitute the training part of the single-stage long-tail target detection scheme of the embodiments of the present application; and the first part 51, the second part 52 and the fourth part 54 together constitute the test part of the single-stage long-tail target detection scheme of the embodiments of the present application.

[0193] The target detection method proposed in the embodiments of the present application is the first long-tail single-stage target detection scheme, which improves the single-stage algorithm ATSS, adds stability settings and enhancement settings, and proposes an enhanced baseline. Moreover, the embodiments of the present application comprehensively consider the sample imbalance among foreground classes and the foreground-background imbalance, and propose a balanced focal loss to simultaneously solve the two problems.

[0194] Therefore, the target detection method in the embodiments of the present application can achieve the following technical effects: 1) closer to the real scene, while having the ability to solve the long-tail problem and the advantages of single-stage detector fast reasoning and easy deployment, having advantages in simplicity and efficiency. 2) Provide an enhanced baseline for single-stage long-tail target detection, facilitating subsequent research. 3) Simultaneously solves the two major problems of sample imbalance between foreground classes and foreground-background imbalance in single-stage long-tail target detection; attributes the two major imbalance problems to the inconsistency of positive and negative sample imbalance degrees of different classes in the long-tail scene, and then proposes two adjustment factors, focus factor and weight factor, to solve the two major imbalance problems. 4) Based on the two adjustment factors, an equilibrium focal loss is proposed, which, combined with the enhanced baseline, has superior performance and surpasses the existing long-tail target detection algorithm. 5) The equilibrium focal loss has low coupling degree with the model and can be applied to most single-stage models.

[0195] Therefore, the user scenarios of the target detection method in the embodiments of the present application include: 1) long-tail target detection task: there are long-tail problems in many real application scenarios, such as autonomous driving, smart city, industrial detection, etc., and there are cases of large sample size of common classes and small sample size of rare classes. The target detection method in the embodiments of the present application can be well combined with these tasks and perform well in long-tail scenarios. 2) Training and deployment process of single-stage detector: since the target detection method in the embodiments of the present application is a long-tail solution based on single-stage models, it is closer to the real scene and has the advantage of easy deployment of single-stage detectors. The target detection method in the embodiments of the present application can be well deployed to downstream devices and is more robust for scene detection in the real world.

[0196] Based on the foregoing embodiments, the embodiments of the present application further provide a training method of a detection model, wherein the training method uses a new loss function, which will be described in detail as follows:

[0197] 1) focal loss;

[0198] In single-stage detectors, focal loss is widely used to solve the foreground-background imbalance problem. Focal loss redistributes the loss contribution of simple samples and difficult samples, thereby weakening most background samples. The traditional focal loss can be obtained by the following formula (7):

[0199] ……………………………(7);

[0200] wherein, the value range of p is 0 to 1, indicating the prediction confidence score of the candidate object; is used to balance positive samples and negative samples. The modulation factor It is a key component of focus loss, reducing the weight of easy samples and increasing the learning focus on difficult samples. Since negative samples are easier to classify than positive samples, the imbalance between positive and negative samples can be viewed as an imbalance between easy and difficult samples. Taking a larger value will significantly reduce the loss contribution of negative samples, thereby improving the impact of positive samples on the training results. In other words, the higher the imbalance between positive and negative samples, the greater the impact of positive samples on the training results. The expected value is larger.

[0201] Because focus loss treats the learning process equally for all classes, and all classes have the same modulation factor, focus loss cannot handle the long-tail imbalance problem.

[0202] 2) Equilibrium focus loss;

[0203] In long-tail datasets, besides foreground-background imbalance, single-stage detectors also suffer from imbalance among foreground classes. Research reveals that rare classes experience more severe positive-negative sample imbalance in long-tail data. That is, most single-stage detectors perform worse on rare classes than on frequent classes, indicating that the same modulation factor is not suitable for all imbalance problems of varying degrees in both positive and negative samples.

[0204] Based on the above analysis, this application proposes to use class-related equilibrium focus loss (EFL) to solve the imbalance problem of different degrees of positive and negative samples. The equilibrium focus loss is class-related, and the equilibrium focus loss of the j-th class can be obtained by the following formula (8):

[0205] ………………………………(8;

[0206] in, and Same as in traditional focus loss; parameters It is the first A type of focus factor, which, in addition to having parameters similar to those in traditional focus loss, In addition to similar functions, it can also be used to alleviate the imbalance problem between positive and negative samples of different categories. The focusing factor... It consists of two components, one of which is a category-independent parameter. A specific parameter related to the category. The focusing factor It can be obtained through the following formula (9):

[0207] …………………………………(9;

[0208] wherein, denotes the parameter in the balanced data that controls the basic behavior of the classifier; , whose value range is greater than or equal to 0, is a parameter related to the balance degree of the class j, which gives a large value to the rare class according to the gradient weighting mechanism proposed in EQLv2, so as to expand the attention to the imbalance of positive and negative samples of the rare class, and gives a small value to the frequent class, so as to maintain the basic behavior of the loss function. denotes the cumulative gradient ratio of the positive and negative samples of the jth class, The larger the value is, the more balanced the jth class is trained, The smaller the value is, the more unbalanced the jth class is trained. In order to meet the requirements of , the present embodiment can stipulate that takes a value in the range of 0 to 1, and adopts to reverse its distribution. The hyperparameter is a proportional factor, which is used to determine the upper limit of in EFL. Compared with focal loss, the EFL proposed in the present embodiment can respectively handle the positive and negative sample imbalance problem of each class, thereby improving the performance of the detector.

[0209] Only the focal factor cannot make the detection model achieve the optimal performance, because in the scene of common classification of multiple classes, The larger the value is, the smaller the loss contribution of the jth class is in the training process. The rare class needs to use a large to learn the extreme positive and negative sample imbalance problem, but it will lose the gradient contribution because of the large . Moreover, when the rare difficult sample and the frequent difficult sample learn together, even if each class has different , the loss contributions of the two are basically the same. In fact, because the number of rare difficult samples is small, the rare difficult sample should have more loss contribution than the frequent difficult sample.

[0210] Therefore, the present embodiment proposes a weight factor to rebalance the loss contribution of different classes to alleviate the above situation. Similar to the focal factor, the present embodiment assigns a large weight factor value to the rare class to improve its loss contribution, and keeps the weight factor value close to 1 for the frequent class. The present embodiment uses to represent the weight factor of the jth class, which is related to the focal factor. The final balanced focal loss can be obtained by the following formula (10):

[0211] ……………………(10);

[0212] That is, the balanced focal loss with the weight factor significantly increases the loss contribution of the rare class and improves the learning focus on the rare difficult samples relative to the frequent difficult samples.

[0213] In summary, the focusing factor and the weight factor constitute the class-related adjustment factor in the balanced focal loss EFL, which enables the detection model to dynamically adjust the loss contribution. The EFL not only improves the learning intensity of the rare class relative to the frequent class, but also balances the loss contribution of the foreground samples and the background samples, and improves the loss contribution of the rare class relative to the frequent class. Further, the EFL can simultaneously solve the foreground class imbalance problem and the foreground-background imbalance problem. Moreover, if the data is balanced distribution, the loss of each class in the EFL is That is, the EFL is equivalent to the focal loss in the case of balanced distribution, and this property enables the EFL to be applied to different distribution data and different data samplers.

[0214] Based on the foregoing embodiments, the embodiments of the present application provide a training device of a detection model, which comprises various units, various sub-units and various modules included in the units, and various sub-modules and various components included in the modules, and can be realized by a processor in an electronic device; of course, it can also be realized by a specific logic circuit; in the implementation process, the processor can be a CPU (Central Processing Unit), MPU (Microprocessor Unit), DSP (Digital Signal Processing) or FPGA (Field Programmable Gate Array) and the like.

[0215] Figure 7 The composition structure diagram of the training device of the detection model of the embodiments of the present application is shown in FIG. 6, which comprises: Figure 7

[0216] A first prediction unit 601 is configured to perform target detection on the obtained training image set by using the detection model to be trained, to obtain a predicted detection result.

[0217] A degree determination unit 602 is configured to determine the imbalance degree of the positive and negative samples of different classes in the training image set.

[0218] A loss determination unit 603 is configured to determine the loss of the predicted detection result in the different classes based on the predicted detection result and the imbalance degree of the positive and negative samples of the different classes.

[0219] ​The parameter updating unit 604 is configured to update network parameters in the detection model to be trained by using the loss until the updated detection model meets a convergence condition.

[0220] In some embodiments, the degree determining unit 602 includes:

[0221] A cumulative gradient ratio determining module is configured to determine cumulative gradient ratios of positive and negative samples of different categories in the training image set.

[0222] A degree determining module is configured to determine imbalance degrees of the positive and negative samples of the different categories based on the cumulative gradient ratios of the positive and negative samples of the different categories.

[0223] In some embodiments, the loss determining unit 603 includes:

[0224] A weighting module is configured to weight the imbalance degrees of the positive and negative samples of the different categories by using a first hyperparameter to obtain a weighting result of each category in the different categories, wherein the first hyperparameter is used to control a learning strength of the detection model to be trained on the imbalance degrees of positive and negative samples of a rare category in the different categories.

[0225] A first loss determining module is configured to determine a loss of the predicted detection result in each category based on the predicted detection result and the weighting result of each category.

[0226] In some embodiments, the loss determining unit 603 includes:

[0227] A focal loss determining module is configured to determine a focal loss of the predicted detection result.

[0228] A second loss determining module is configured to adjust the focal loss by using the imbalance degrees of the positive and negative samples of the different categories to obtain the loss of the predicted detection result in the different categories.

[0229] In some embodiments, the different categories at least include a rare category and a frequent category, and the apparatus further includes:

[0230] A weight determining unit is configured to determine a weight factor based on a basic factor and the imbalance degrees of the positive and negative samples of the rare category and the frequent category, wherein the basic factor is used to balance loss contributions of foreground regions and background regions in the training image.

[0231] An optimization unit is configured to adjust a loss contribution of the rare category relative to the frequent category in the loss by using the weight factor to obtain an optimized loss.

[0232] The parameter updating unit 604 includes:

[0233] The first parameter updating sub-unit is configured to update network parameters in the detection model to be trained by using the optimized loss.

[0234] In some embodiments, the parameter updating unit 604 includes:

[0235] The first gradient determining module is configured to determine a gradient value output by the detection model to be trained in an updating process based on the loss.

[0236] The second gradient determining module is configured to perform gradient clipping on the gradient value to obtain a clipped gradient value in a case where the gradient value is greater than or equal to a preset threshold.

[0237] The parameter updating module is configured to update network parameters in the detection model based on the clipped gradient value.

[0238] In some embodiments, the apparatus further includes:

[0239] The iteration number determining unit is configured to determine an iteration number of iterative training of the detection model to be trained in a pre-training stage based on a distribution of images in the training image set.

[0240] The pre-training unit is configured to pre-train the detection model to be trained based on the iteration number and the training image set to obtain a candidate detection model.

[0241] The parameter updating unit 604 includes:

[0242] The second parameter updating sub-unit is configured to update network parameters in the candidate detection model by using the loss until the updated detection model satisfies a convergence condition.

[0243] In some embodiments, the iteration number determining unit includes:

[0244] The first iteration number determining module is configured to set the iteration number of the detection model to be trained in the pre-training stage to a first preset value in a case where the training image set is a long-tail distribution image set.

[0245] The second iteration number determining module is configured to set the iteration number to a second preset value in a case where the training image set is a non-long-tail distribution image set; and the first preset value is greater than the second preset value.

[0246] In some embodiments, the first prediction unit 601 includes:

[0247] The prediction box generation module is configured to generate at least two prediction boxes on each region of the training image by using the detection model to be trained, wherein the at least two prediction boxes have different scales.

[0248] The intersection-over-union determination module is configured to determine an intersection-over-union between the generated prediction box and the labeled box of the training image.

[0249] The prediction module is configured to measure a probability that a target in the prediction box is foreground based on the intersection-over-union, to obtain a detection result of the training image prediction.

[0250] In some embodiments, the apparatus further includes:

[0251] The preprocessing unit is configured to pre-process each training image in the training image set to obtain a pre-processed training image.

[0252] The first feature extraction unit is configured to perform feature extraction on the pre-processed training image to obtain a feature map.

[0253] The prediction box generation module includes:

[0254] The prediction box generation submodule is configured to generate at least two prediction boxes on each pixel of the feature map by using the detection model to be trained, wherein the pixels of the feature map have a corresponding relationship with the regions in the training image.

[0255] Based on the foregoing embodiments, the embodiments of the present application provide a target detection apparatus, which includes various units, sub-units, modules, sub-modules and components included in each unit, and can be implemented by a processor in an electronic device. Of course, it can also be implemented by a specific logic circuit. In the implementation process, the processor can be a CPU, MPU, DSP or FPGA, etc.

[0256] Figure 8 The target detection apparatus of the embodiments of the present application is a component structure schematic diagram, as shown in Figure 8 The target detection apparatus 700 includes:

[0257] The acquisition unit 701 is configured to acquire a to-be-detected image.

[0258] The preprocessing unit 702 is configured to pre-process the to-be-detected image to obtain a pre-processed to-be-detected image.

[0259] The feature extraction unit 703 is configured to perform feature extraction on the pre-processed to-be-detected image to obtain a feature map of the to-be-detected image.

[0260] The second prediction unit 704 is configured to input the feature map of the image to be detected into a detection model to obtain a prediction result, wherein the detection model is trained based on the training method of the detection model provided in the above embodiments.

[0261] The post-processing unit 705 is configured to post-process the prediction result to obtain a detection result of the image to be detected.

[0262] In some embodiments, the prediction result includes a plurality of prediction boxes and a target category corresponding to each prediction box; and the post-processing unit 705 includes:

[0263] The post-processing sub-unit is configured to filter the prediction boxes with overlap in the plurality of prediction boxes to obtain filtered prediction boxes and a target category corresponding to each filtered prediction box.

[0264] The above device embodiments are similar to the descriptions of the above method embodiments, and have similar beneficial effects to the method embodiments. For technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0265] It should be noted that, in the embodiments of the present application, if the training method or the detection method described above is realized in the form of a software function module and is sold or used as an independent product, it can also be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing an electronic device (which can be a personal computer, a server, etc.) to execute all or part of the methods described in the embodiments of the present application. The storage medium mentioned above includes: a U disk, a mobile hard disk, a ROM (Read Only Memory), a magnetic disk or an optical disk, and various storage media that can store program codes. Thus, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0266] Correspondingly, the embodiments of the present application provide an electronic device, which includes a memory and a processor, the memory stores a computer program capable of running on the processor, and the processor executes the program to realize the steps in the training method or the steps in the detection method provided in the above embodiments.

[0267] Correspondingly, the embodiments of the present application provide a readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the steps in the training method or the steps in the detection method.

[0268] It should be noted that the above description of the storage medium and platform embodiments is similar to the description of the method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and platform embodiments of the present application, please refer to the description of the method embodiments for understanding.

[0269] It should be noted that, ​ A hardware entity diagram of an electronic device according to an embodiment of the present application is shown in FIG. 8, which includes a processor 801, a communication interface 802 and a memory 803, wherein ​ The processor 801 generally controls the overall operation of the electronic device 800.

[0270] The processor 801 generally controls the overall operation of the electronic device 800.

[0271] The communication interface 802 can enable the electronic device 800 to communicate with other servers or electronic devices or platforms through a network.

[0272] The memory 803 is configured to store instructions and applications executable by the processor 801, and can also cache data to be processed by the processor 801 and modules in the electronic device 800 (for example, image data, audio data, voice communication data and video communication data), which can be implemented by FLASH (Flash Memory) or RAM (Random Access Memory).

[0273] The various hardware entities in the electronic device 800 are coupled together through a bus 804. It can be understood that the bus 804 is used to realize the connection and communication between the hardware entities.

[0274] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division, and actual implementation can have another division manner, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the various components shown or discussed can be through some interface, indirect coupling or communication connection between devices or units, which can be electrical, mechanical or other forms.

[0275] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or distributed on multiple network units; some or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment.

[0276] In addition, all the functional units in the embodiments of the present application can be integrated in one processing module, or each unit can be separately taken as one unit, or two or more units can be integrated in one unit; the integrated unit can be realized in the form of hardware or in the form of hardware plus software function unit. Those skilled in the art can understand that all or part of the steps of the foregoing method embodiments can be completed by a program instructing related hardware, and the foregoing program can be stored in a computer readable storage medium, and when the program is executed, the steps of the foregoing method embodiments are executed; and the foregoing storage medium includes: mobile storage equipment, ROM, RAM, magnetic disk or optical disk and various storage medium capable of storing program codes.

[0277] The methods disclosed in the several method embodiments provided by the present application can be combined arbitrarily without conflict, to obtain new method embodiments. The features disclosed in the several product embodiments provided by the present application can be combined arbitrarily without conflict, to obtain new product embodiments. The features disclosed in the several method or device embodiments provided by the present application can be combined arbitrarily without conflict, to obtain new method embodiments or device embodiments.

[0278] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited to this, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for training a detection model, characterized in that, The method includes: The detection model to be trained is used to perform target detection on the acquired training image set to obtain the predicted detection results; the detection results include the detected targets and the target category of each target; Determine the degree of imbalance between positive and negative samples of different categories in the training image set; positive samples are foreground samples, and negative samples are background samples. Based on the predicted detection results and the degree of imbalance between positive and negative samples of different categories, the loss of the predicted detection results in the different categories is determined; The network parameters in the detection model to be trained are updated using the loss until the updated detection model meets the convergence condition.

2. The training method according to claim 1, characterized in that, Determining the imbalance of positive and negative samples of different categories in the training image set includes: Determine the cumulative gradient ratio of positive and negative samples of different categories in the training image set; The degree of imbalance between positive and negative samples of different categories is determined based on the cumulative gradient ratio of the positive and negative samples of different categories.

3. The method according to claim 1, characterized in that, The determination of the loss of the predicted detection result in the different categories based on the imbalance between the predicted detection result and the positive and negative samples of the different categories includes: The imbalance of positive and negative samples of different categories is weighted using a first hyperparameter to obtain the weighted result of each category; wherein, the first hyperparameter is used to control the strength of the detection model to be trained in learning the imbalance of positive and negative samples of rare categories in different categories. Based on the predicted detection results and the weighted results for each category, the loss of the predicted detection results in each category is determined.

4. The method according to any one of claims 1 to 3, characterized in that, The determination of the loss of the predicted detection result in the different categories based on the imbalance between the predicted detection result and the positive and negative samples of the different categories includes: Determine the focus loss of the predicted detection result; The focus loss is adjusted by the degree of imbalance between positive and negative samples of different categories to obtain the loss of the predicted detection result in the different categories.

5. The method according to claim 4, characterized in that, The different categories include at least: rare categories and frequent categories. After adjusting the focus loss based on the imbalance of positive and negative samples in the different categories to obtain the loss of the predicted detection result in the different categories, the method further includes: Weighting factors are determined based on the basic factors and the imbalance between positive and negative samples of the rare and frequent categories; wherein the basic factors are used to balance the loss contributions of the foreground and background regions in the training image. Based on the weighting factor, the loss contribution of rare classes relative to frequent classes in the loss is adjusted to obtain the optimized loss; The step of updating the network parameters in the detection model to be trained using the loss includes: updating the network parameters in the detection model to be trained using the optimized loss.

6. The method according to any one of claims 1 to 3, characterized in that, The step of updating the network parameters in the detection model to be trained using the loss includes: Based on the loss, determine the gradient value output by the detection model to be trained during the update process; If the gradient value is greater than or equal to a preset threshold, the gradient value is clipped to obtain the clipped gradient value. Based on the cropped gradient values, the network parameters in the detection model are updated.

7. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Based on the distribution of images in the training image set, the number of iterations for iterative training of the detection model to be trained in the pre-training stage is determined. Based on the number of iterations and the training image set, the detection model to be trained is pre-trained to obtain a candidate detection model; The step of updating the network parameters in the detection model to be trained using the loss until the updated detection model satisfies the convergence condition includes: The network parameters in the candidate detection model are updated using the loss until the updated detection model meets the convergence condition.

8. The method according to claim 7, characterized in that, The step of determining the number of iterations for iterative training of the detection model to be trained during the pre-training phase, based on the distribution of images in the training image set, includes: In the case that the training image set is a long-tailed image set, the number of iterations of the detection model to be trained in the pre-training stage is set to a first preset value; When the training image set is a non-long-tailed image set, the number of iterations is set to a second preset value; wherein the first preset value is greater than the second preset value.

9. The method according to any one of claims 1 to 3, characterized in that, The step involves using a detection model to be trained to perform object detection on the acquired training image set, obtaining predicted detection results, including: Using a detection model to be trained, at least two prediction boxes are generated in each region of the training image; wherein the at least two prediction boxes have different scales. Determine the intersection ratio (IoU) between the generated predicted bounding box and the labeled bounding box of the training image; Based on the intersection ratio, the probability that the target in the prediction box is the foreground is measured to obtain the detection result of the training image prediction.

10. The method according to claim 9, characterized in that, The method further includes: Each training image in the training image set is preprocessed to obtain a preprocessed training image. Feature extraction is performed on the preprocessed training images to obtain feature maps; The method employs a detection model to be trained, generating at least two prediction boxes in each region of the training image, including: Using the detection model to be trained, at least two prediction boxes are generated on each pixel of the feature map; wherein the pixels of the feature map correspond to the regions in the training image.

11. A target detection method, characterized in that, The method includes: Acquire the image to be detected; The image to be detected is preprocessed to obtain the preprocessed image to be detected; Feature extraction is performed on the preprocessed image to be detected to obtain a feature map of the image to be detected; The feature map of the image to be detected is input into the detection model to obtain the prediction result; wherein the detection model is trained based on the method described in any one of claims 1 to 10; The prediction results are post-processed to obtain the detection results of the image to be detected.

12. The method according to claim 11, characterized in that, The prediction results include multiple prediction boxes and the target category corresponding to each prediction box; The post-processing of the prediction result to obtain the detection result of the image to be detected includes: The overlapping prediction boxes among the multiple prediction boxes are filtered to obtain the filtered prediction boxes and the target category corresponding to each filtered prediction box.

13. A training device for a detection model, characterized in that, The device includes: The first prediction unit is used to perform target detection on the acquired training image set using the detection model to be trained, and to obtain the predicted detection result; the detection result includes the detected target and the target category of each target; The degree determination unit is used to determine the degree of imbalance between positive and negative samples of different categories in the training image set; positive samples are foreground samples and negative samples are background samples. A loss determination unit is used to determine the loss of the predicted detection result in the different categories based on the predicted detection result and the degree of imbalance between the positive and negative samples of the different categories; The parameter update unit is used to update the network parameters in the detection model to be trained using the loss until the updated detection model meets the convergence condition.

14. A target detection device, characterized in that, The device includes: The acquisition unit is used to acquire the image to be detected; The preprocessing unit is used to preprocess the image to be detected to obtain the preprocessed image to be detected. The feature extraction unit is used to extract features from the preprocessed image to be detected to obtain a feature map of the image to be detected. The second prediction unit is used to input the feature map of the image to be detected into the detection model to obtain the prediction result; wherein the detection model is trained based on the method described in any one of claims 1 to 10; The post-processing unit is used to post-process the prediction results to obtain the detection results of the image to be detected.

15. An electronic device comprising a memory and a processor, the memory storing a computer program executable on the processor, the processor executing the program to implement the steps of the method of any one of claims 1 to 10, or to implement the steps of the method of any one of claims 11 to 12.

16. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10, or implements the steps of the method according to any one of claims 11 to 12.

Images