Joint optimization method, device, equipment and medium for small target detection

Abstract

The present invention relates to a joint optimization method, device, equipment and medium for small target detection. The method comprises: obtaining a training sample set, inputting the training sample set into a detector, obtaining a classification output result and a bounding box regression output result of each training sample; calculating a classification score of each training sample according to the classification label and the classification output result; calculating a bounding box regression score of each training sample according to the true value box and the bounding box regression output result; calculating a positive sample weight and a negative sample weight according to the classification score and the bounding box regression score; then obtaining a joint optimization loss according to the positive sample weight and the negative sample weight; back-propagating the joint optimization loss to the detector for optimization training to obtain a trained detector, and performing small target detection through the trained detector. The present invention can well balance classification prediction and regression prediction, and improve the detection rate of small targets.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a joint optimization method, device, equipment and medium for small target detection. Background Art

[0002] With the development of technology, artificial intelligence (AI) applications have gradually penetrated into all aspects of daily life in recent years, among which object detection has become one of the most widely used technologies. Compared with traditional methods, the emergence of neural networks and deep learning has significantly improved the accuracy and processing speed of detection. In particular, the development of residual networks has made it possible to efficiently train deeper and more complex neural networks, resulting in a significant improvement in accuracy. Recently proposed detectors such as Co-DETR and MoCaE have demonstrated state-of-the-art performance on the MS COCO 2017 dataset.

[0003] Although general object detection technology has made significant progress, small object detection is still a major problem in the field of computer vision. Some literature points out that there are three main challenges in small object detection: information loss, noise interference, and insufficient training samples. Therefore, when general object detectors are directly applied to small object detection tasks, the accuracy rate often decreases significantly. After analysis, it is found that there are two key factors in small object detection that have a great impact on the detection effect.

[0004] First of all, as we all know, the core process of model training is back propagation, which directly affects parameter updates. The basis of this process is the loss function of the model. Therefore, for targets with a larger proportion of loss value in the total loss function, the model tends to accurately predict them. This is also one of the main reasons why the detection accuracy of small targets is much lower than that of large targets.

[0005] Secondly, most existing detectors directly add the classification loss and the bounding box regression loss during training, and the weights of the classification loss and the bounding box regression loss of each sample are both 1. Some methods for improving the loss function mostly focus on only one of the classification loss and the regression loss. For example, Focal Loss only improves the classification loss function without optimizing the bounding box regression loss function, and IoU Loss only improves the bounding box regression loss function without modifying the classification loss. This may lead to an imbalance between classification prediction and regression prediction, such as some predicted samples are accurately positioned but incorrectly classified, or some predicted samples are accurately classified but inaccurately positioned. Summary of the invention

[0006] Based on this, it is necessary to provide a joint optimization method, device, equipment and medium for small target detection that can solve the problems of low small target detection rate and imbalance between classification prediction and regression prediction in response to the above-mentioned technical problems.

[0007] A joint optimization method for small target detection, the method comprising:

[0008] Obtaining a training sample set, inputting the training sample set into a detector, and obtaining a classification output result and a bounding box regression output result of each training sample;

[0009] Calculating a classification score for each training sample based on the classification label and the classification output result; calculating a bounding box regression score for each training sample based on the true value box and the bounding box regression output result;

[0010] Calculating a positive sample weight and a negative sample weight according to the classification score and the bounding box regression score; and then obtaining a joint optimization loss according to the positive sample weight and the negative sample weight;

[0011] The joint optimization loss is back-propagated to the detector for optimization training to obtain a trained detector, and small target detection is performed using the trained detector.

[0012] A joint optimization device for small target detection, the device comprising:

[0013] An output module is used to obtain a training sample set, input the training sample set into the detector, and obtain a classification output result and a bounding box regression output result of each training sample;

[0014] A score calculation module, used to calculate the classification score of each training sample according to the classification label and the classification output result; and calculate the bounding box regression score of each training sample according to the true value box and the bounding box regression output result;

[0015] A joint optimization loss calculation module, used to calculate the positive sample weight and the negative sample weight according to the classification score and the bounding box regression score; and then obtain the joint optimization loss according to the positive sample weight and the negative sample weight;

[0016] The optimization training module is used to perform optimization training on the detector by back-propagating the joint optimization loss to obtain a trained detector, and perform small target detection by the trained detector.

[0017] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the joint optimization method for small target detection when executing the computer program.

[0018] A computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of the joint optimization method for small target detection are implemented.

[0019] The above-mentioned joint optimization method, device, equipment and medium for small target detection obtains a training sample set, inputs the training sample set into the detector, and obtains the classification output result and bounding box regression output result of each training sample; calculates the classification score of each training sample according to the classification label and the classification output result; calculates the bounding box regression score of each training sample according to the true value box and the bounding box regression output result; calculates the positive sample weight and the negative sample weight according to the classification score and the bounding box regression score; then obtains the joint optimization loss according to the positive sample weight and the negative sample weight; back-propagates the joint optimization loss to the detector for optimization training to obtain a trained detector, and performs small target detection through the trained detector.

[0020] The present invention evaluates the prediction quality of each sample by calculating the classification score and bounding box regression score of each training sample. Then, the positive sample weight and the negative sample weight are calculated based on the classification score and bounding box regression, so as to dynamically assign appropriate weights to each training sample, give higher weights to samples with inaccurate predictions, balance classification predictions and regression predictions, and make the detector pay more attention to these challenging training samples. The method proposed in the present invention can well improve the detection rate of small targets, especially for tiny targets, there is a significant improvement in the accuracy rate; in addition, it can be well integrated into most existing general target detectors and loss functions, and there is no need to modify the detection network architecture or add additional hyperparameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0022] Figure 1 A schematic diagram of a process flow of a joint optimization method for small target detection in one embodiment;

[0023] Figure 2 A schematic diagram of a framework of a joint optimization method for small target detection in one embodiment;

[0024] Figure 3 This is a schematic diagram of the comparison of the first group of detection effects in an embodiment. Figure 3 (a) is a schematic diagram of the detection results obtained using the reference detector. Figure 3 (b) is a schematic diagram of the detection results obtained by using the joint optimization method proposed in the present invention;

[0025] Figure 4This is a schematic diagram of the comparison of the first group of detection effects in an embodiment. Figure 4 (a) is a schematic diagram of the detection results obtained using the reference detector. Figure 4 (b) is a schematic diagram of the detection results obtained by using the joint optimization method proposed in the present invention;

[0026] Figure 5 It is a structural block diagram of a joint optimization device for small target detection in one embodiment;

[0027] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment.

[0028] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] It should be noted that the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0031] The following will describe the implementation of the present invention in detail with reference to the accompanying drawings in the embodiment diagram of the present invention.

[0032] Example 1

[0033] This embodiment discloses a joint optimization method for small target detection, which is used to evaluate the prediction quality of each sample by calculating the classification score and bounding box regression score of each training sample. Then, the positive sample weight and the negative sample weight are calculated based on the classification score and bounding box regression, so as to dynamically assign appropriate weights to each training sample, give higher weights to samples with inaccurate predictions, balance classification predictions and regression predictions, and make the detector pay more attention to these challenging training samples. The method proposed in the present invention can well improve the detection rate of small targets, especially for tiny targets, there is a significant improvement in the accuracy rate; in addition, it can be well integrated into most existing general target detectors and loss functions, and there is no need to modify the detection network architecture or add additional hyperparameters.

[0034] like Figure 1and 2 As shown, the joint optimization method for small target detection provided in this embodiment includes the following steps:

[0035] Step 201 , obtain a training sample set, input the training sample set into a detector, and obtain a classification output result and a bounding box regression output result of each training sample.

[0036] Step 202, calculating the classification score of each training sample according to the classification label and the classification output result; calculating the bounding box regression score of each training sample according to the true value box and the bounding box regression output result.

[0037] Step 203, calculate the positive sample weight and the negative sample weight according to the classification score and the bounding box regression score; then obtain the joint optimization loss according to the positive sample weight and the negative sample weight.

[0038] Step 204, optimize and train the detector by back-propagating the joint optimization loss to obtain a trained detector, and perform small target detection using the trained detector.

[0039] In the specific implementation process of step 201, the training sample set is a data set for training the detector, including image data and target annotation data, such as the commonly used AI-TOD data set. The detector is a general basic detector, such as the two-stage anchor-based detector Faster R-CNN, Cascade R-CNN and DetectoRS, the single-stage anchor-based detector RetinaNet, and the anchor-free detector FCOS, etc. The joint optimization method for small target detection proposed in the present invention is applied to the general basic detector for training optimization to improve the detection rate of small targets, especially for small targets. There is a significant improvement in the accuracy. In addition, the method proposed in the present invention can also be well combined with commonly used classification loss functions, such as Cross Entropy Loss, Focal Loss, and bounding box regression loss functions L1Loss, L2 Loss, IoU Loss, etc.

[0040] Inputting the training sample set into the detector to obtain the classification output result and bounding box regression output result of each training sample is the basic process of the detector and will not be described in detail here.

[0041] In the specific real-time process of step 202, the target annotation data in the training sample set includes classification labels and true value boxes. The classification score of each training sample is calculated by the classification label and the classification output result. , the bounding box regression score of each training sample is calculated by the true value box and the bounding box regression output ,These two scores are used to evaluate the prediction quality of each sample and improve the detection accuracy.

[0042] In the specific real-time process of step 203, the positive sample weight is calculated according to the classification score and the bounding box regression score, including: determining the weight value range limit value according to the required positive sample weight value range; calculating the positive sample weight according to the weight value range limit value, the classification score and the bounding box regression score.

[0043] The negative sample weight is calculated according to the classification score and the bounding box regression score, including: obtaining a positive sample identifier; and calculating the average value of the positive sample weight according to the positive sample identifier to obtain the negative sample weight.

[0044] A joint optimization loss function is obtained according to the positive sample weights and the negative sample weights, including: obtaining a negative sample identifier; obtaining a basic classification loss according to a basic classification loss function, and obtaining a bounding box regression basic loss according to a basic bounding box regression loss function; and obtaining a joint optimization loss of training samples by calculating according to the positive sample identifier, the negative sample identifier, the positive sample weights, the negative sample weights, the basic classification loss, and the bounding box regression basic loss.

[0045] Specifically, the calculation expression of the positive sample weight is:

[0046] ;

[0047] The calculation expression of negative sample weight is:

[0048] ;

[0049] The calculation expression of the joint optimization loss is:

[0050] ;

[0051] In the formula, represents the weight of positive samples; Indicates Bounding box regression scores for training samples; Indicates The classification scores of training samples are Indicates the value limit of the weight value range. Represents the weight of negative samples; Represents the positive sample identifier; represents the training sample set; represents training samples; Indicates The bounding box regression base loss for training samples; Indicates The basic classification loss of training samples; Represents the negative sample identifier; represents the weight of negative samples.

[0052] It can be understood that usually according to the classification loss function and the bounding box regression loss function Get the first The basic classification loss of training samples and bounding box regression base loss . Basic classification loss function and the basic bounding box regression loss function Choose according to the type of detector, such as in Faster R-CNN, the bounding box regression loss function is L1 loss, classification loss function is the cross entropy loss.

[0053] According to the calculation mechanism of the positive sample weight and the negative sample weight, the most appropriate weight for each training sample can be dynamically calculated, where the positive sample weight is used to focus on the area that matches the target category, and this value varies with the different prediction effects of the positive samples; the negative sample weight is the average of the weights of all positive samples in the batch, which is mainly used to balance the positive sample weight and the negative sample weight.

[0054] Then the calculated weight of each training sample is multiplied by the base loss of each training sample to obtain the joint optimization loss.

[0055] In one of the embodiments, in order to intuitively compare the improvement in detection accuracy brought about by embedding the method proposed in the present invention into a commonly used basic detector, the most commonly used two-stage anchor frame-based target detector Faster R-CNN is used as a benchmark detector, and training and testing are performed on the commonly used remote sensing small target detection dataset AI-TOD. The detection effect of the benchmark detector and the detection effect after applying the joint optimization method for small target detection proposed in the present invention to this benchmark detector are compared. In addition to using the method proposed in the present invention, other parameter configurations of the model during training and testing are consistent with the default configuration of the benchmark detector.

[0056] Detection effect comparison Figure 3 and Figure 4 As shown, the green, blue and red rectangular boxes represent the target objects that are correctly detected, falsely detected and missed, respectively. By comparison, it can be intuitively seen that the method proposed in the present invention brings a significant improvement in the accuracy rate.

[0057] In addition to the visual comparison, the present invention also conducts a detailed experimental comparison on the AI-TOD dataset. The specific results are shown in Table 1, wherein rows 1 to 5 are the results on a two-stage anchor-frame-based target detector, rows 6 to 9 are the detection results using a one-stage anchor-frame-based target detector, rows 10 to 13 are the detection results obtained using a detector without an anchor frame, rows 14 to 17 are the detection results of the four existing methods with the highest accuracy on the AI-TOD dataset, and rows 18 to 22 are the detection results of the method proposed in the present invention applied to a commonly used general target detector.

[0058] The method proposed in the present invention can be easily adapted to various general target detectors. As can be seen from Table 1, after using the method proposed in the present invention on the benchmark detectors RetinaNet, FCOS, Faster R-CNN, Cascade R-CNN and DetectoRS, the detection accuracy can be improved by 6.4, 5.2, 13.5, 11.7 and 12.4 percentage points respectively. Compared with the current method with the highest accuracy on AI-TOD, the method of the present invention can bring an accuracy improvement of 0.6 percentage points.

[0059] Table 1 Comparison of detection accuracy on the AI-TOD test set

[0060]

[0061] It can be seen that the joint optimization loss function for small target detection proposed in the present invention can well improve the detection effect of small target detection tasks, especially for those target objects with very small sizes, the detection accuracy improvement brought by the method proposed in the present invention is more prominent.

[0062] Although this embodiment Figure 1 The steps in the process are shown in sequence as indicated by the arrows, but the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of the steps, and the steps can be executed in other orders. Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0063] Example 2

[0064] Based on the joint optimization method for small target detection in Example 1, this embodiment discloses a joint optimization device for small target detection, such as Figure 5 As shown, the joint optimization device for small target detection includes: an output module 401, a score calculation module 402, a joint optimization loss calculation module 403 and an optimization training module 404, wherein:

[0065] The output module 401 is used to obtain a training sample set, input the training sample set into the detector, and obtain a classification output result and a bounding box regression output result of each training sample.

[0066] The score calculation module 402 is used to calculate the classification score of each training sample according to the classification label and the classification output result; and calculate the bounding box regression score of each training sample according to the true value box and the bounding box regression output result.

[0067] The joint optimization loss calculation module 403 is used to calculate the positive sample weight and the negative sample weight according to the classification score and the bounding box regression score; and then obtain the joint optimization loss according to the positive sample weight and the negative sample weight.

[0068] The optimization training module 404 is used to perform optimization training on the detector by back-propagating the joint optimization loss to obtain a trained detector, and perform small target detection by using the trained detector.

[0069] In this embodiment, the specific working process and working principle of the output module 401, the score calculation module 402, the joint optimization loss calculation module 403 and the optimization training module 404 are the same as those in the method of embodiment 1, so they are not described in detail in this embodiment. Each unit module can be implemented in whole or in part by software, hardware and a combination thereof, and each unit module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above unit modules.

[0070] Example 3

[0071] like Figure 6 The terminal device disclosed in this embodiment includes a transmitter, a receiver, a memory, and a processor. The transmitter is used to send instructions and data, the receiver is used to receive instructions and data, the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions stored in the memory to implement the method in the above-mentioned embodiment 1.

[0072] It should be noted that the above memory can be independent or integrated with the processor. When the memory is independently provided, the terminal device further includes a bus for connecting the memory and the processor.

[0073] Example 4

[0074] This embodiment discloses a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method in the above-mentioned embodiment 1 is implemented.

[0075] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0076] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0077] The above-mentioned embodiments only express several implementation modes of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

Claims

1. A joint optimization method for small target detection, characterized in that: The method comprises: Obtaining a training sample set, inputting the training sample set into a detector, and obtaining a classification output result and a bounding box regression output result of each training sample; Calculating a classification score for each training sample based on the classification label and the classification output result; calculating a bounding box regression score for each training sample based on the true value box and the bounding box regression output result; Calculating a positive sample weight and a negative sample weight according to the classification score and the bounding box regression score; and then obtaining a joint optimization loss according to the positive sample weight and the negative sample weight; The joint optimization loss is back-propagated to the detector for optimization training to obtain a trained detector, and small target detection is performed by the trained detector; Calculating the positive sample weight according to the classification score and the bounding box regression score includes: Determine the weight range limit value according to the required positive sample weight range; Calculate the positive sample weight according to the weight value range limit value, the classification score and the bounding box regression score; The calculation expression of the positive sample weight is: ； In the formula, represents the weight of positive samples; Indicates Bounding box regression scores for training samples; Indicates The classification scores of training samples are Indicates the weight value range limit; Calculating a negative sample weight according to the classification score and the bounding box regression score includes: Obtain a positive sample identifier; and calculate an average value of the positive sample weight according to the positive sample identifier to obtain a negative sample weight.

2. The joint optimization method for small target detection according to claim 1, characterized in that: The calculation expression of the negative sample weight is: ； In the formula, Represents the weight of negative samples; Represents the positive sample identifier; represents the training sample set; represents training samples; represents the weight of positive samples.

3. The joint optimization method for small target detection according to claim 1, characterized in that: A joint optimization loss is obtained according to the positive sample weight and the negative sample weight, including: Get negative sample identifier; The basic classification loss is calculated according to the basic classification loss function, and the bounding box regression basic loss is calculated according to the basic bounding box regression loss function; The joint optimization loss of the training samples is calculated based on the positive sample identifier, negative sample identifier, positive sample weight, negative sample weight, basic classification loss and bounding box regression basic loss.

4. The joint optimization method for small target detection according to claim 3, characterized in that: The calculation expression of the joint optimization loss is: ； In the formula, Represents the positive sample identifier; represents the weight of positive samples; Indicates The bounding box regression base loss for training samples; Indicates The basic classification loss of training samples; Represents the negative sample identifier; represents the weight of negative samples.

5. A joint optimization device for small target detection, characterized in that: The joint optimization method for small target detection according to any one of claims 1 to 4 is adopted, wherein the device comprises: An output module is used to obtain a training sample set, input the training sample set into the detector, and obtain a classification output result and a bounding box regression output result of each training sample; A score calculation module, used to calculate the classification score of each training sample according to the classification label and the classification output result; and calculate the bounding box regression score of each training sample according to the true value box and the bounding box regression output result; A joint optimization loss calculation module, used to calculate the positive sample weight and the negative sample weight according to the classification score and the bounding box regression score; and then obtain the joint optimization loss according to the positive sample weight and the negative sample weight; The optimization training module is used to perform optimization training on the detector by back-propagating the joint optimization loss to obtain a trained detector, and perform small target detection by the trained detector.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the joint optimization method for small target detection described in any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the joint optimization method for small target detection described in any one of claims 1 to 4 are implemented.

Images