Target detection method, apparatus, device, and storage medium

Abstract

The application discloses a target detection method, device and equipment and a storage medium. The method comprises the following steps: inputting a training image into a backbone network of a target detection network to perform feature extraction on the training image; inputting features output by the backbone network into a first branch and a first detection head, a second branch and a second detection head of a bottleneck layer to perform processing, so as to obtain a first predicted target frame and a second predicted target frame; determining the size relationship between a first target loss value and a second target loss value based on a preset loss function, and determining a target detection model with a smaller target loss value as a teacher model and a target detection model with a larger target loss value as a student model; and performing joint training on the teacher model and the student model based on a distillation mode, so as to obtain a trained target detection model, thereby detecting a target object in a to-be-detected image. A target detection model with higher accuracy can be obtained, and the target detection model is used for detecting a target object in a to-be-detected image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and in particular, to a target detection method and device, equipment and a storage medium. BACKGROUND

[0002] With the wide application of deep learning target detection methods in various industrial fields, the target detection network has also been relatively rapidly developed. The accuracy of the target detection model is getting higher and higher, but there are still problems such as model complexity and long training period, which affect the efficiency of subsequent target detection.

[0003] The existing target detection model adopts a single processing method in the feature processing part of the bottleneck layer and the target detection part of the detection head, so that the target detection accuracy of the model obtained by training is insufficient.

[0004] Therefore, how to comprehensively use different bottleneck layer processing methods and different detection head target detection methods for model training makes the distillation training process more intelligent, so as to obtain a target detection model with higher accuracy and improve the target detection capability of the image, which is a problem to be solved at present. SUMMARY

[0005] The present application provides a target detection method, device, equipment and storage medium, which can obtain a target detection model with higher accuracy and improve the target detection capability of the product.

[0006] According to an aspect of the present application, a target detection method is provided, comprising:

[0007] inputting a training image into a backbone network of a target detection network to perform feature extraction on the training image;

[0008] inputting the features output by the backbone network into a first branch of a bottleneck layer and a first detection head for processing to obtain a first predicted target frame; and inputting the features output by the backbone network into a second branch of the bottleneck layer and a second detection head for processing to obtain a second predicted target frame;

[0009] based on a preset loss function, determining the size relationship between the first target loss value and the second target loss value according to the first predicted target frame, the second predicted target frame and the real target frame, and determining the target detection model with the smaller target loss value as the teacher model and the target detection model with the larger target loss value as the student model;

[0010] based on a distillation method, jointly training the teacher model and the student model to obtain a trained target detection model, and using the trained target detection model to detect the target object in the image to be detected.

[0011] According to another aspect of the present application, there is provided a target detection device, comprising:

[0012] an extraction module configured to input a training image into a backbone network of a target detection network, and perform feature extraction on the training image;

[0013] a obtaining module configured to input the features output by the backbone network into a first branch and a first detection head of a bottleneck layer for processing to obtain a first predicted target frame, and input the features output by the backbone network into a second branch and a second detection head of the bottleneck layer for processing to obtain a second predicted target frame;

[0014] a determining module configured to determine a size relationship between a first target loss value and a second target loss value based on a preset loss function, according to the first predicted target frame, the second predicted target frame and a real target frame, and determine a target detection model with a smaller target loss value as a teacher model and a target detection model with a larger target loss value as a student model;

[0015] a detection module configured to jointly train the teacher model and the student model in a distillation manner to obtain a trained target detection model, and detect a target object in a to-be-detected image by using the trained target detection model.

[0016] According to another aspect of the present application, there is provided an electronic device, comprising:

[0017] at least one processor; and

[0018] a memory connected in communication with the at least one processor; wherein

[0019] the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the target detection method according to any one of the embodiments of the present application.

[0020] According to another aspect of the present application, there is provided a computer readable storage medium storing computer instructions for enabling a processor to perform the target detection method according to any one of the embodiments of the present application when executed by the processor.

[0021] The technical scheme of the embodiment of the present application inputs a training image into a backbone network of a target detection network, and performs feature extraction on the training image; inputs the features output by the backbone network into a first branch of a bottleneck layer and a first detection head for processing, to obtain a first predicted target frame; and inputs the features output by the backbone network into a second branch of the bottleneck layer and a second detection head for processing, to obtain a second predicted target frame; based on a preset loss function, according to the first predicted target frame, the second predicted target frame and a real target frame, the size relationship of a first target loss value and a second target loss value is determined, and a target detection model with a smaller target loss value is determined as a teacher model, and a target detection model with a larger target loss value is determined as a student model; based on a distillation mode, the teacher model and the student model are jointly trained, to obtain a trained target detection model, and the trained target detection model is used to detect a target object in a to-be-detected image. In this way, different detection heads and bottleneck layer branches can be added for model training, the process of distillation training is more intelligent, and therefore a target detection model with higher accuracy is obtained, which is used to detect a target object in a to-be-detected image.

[0022] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0024] Figure 1 is a flowchart of a target detection method provided by an embodiment of the present application;

[0025] Figure 2 is a flowchart of a target detection method provided by an embodiment of the present application;

[0026] Figure 3 is a structural block diagram of a target detection device provided by an embodiment of the present application;

[0027] Figure 4 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0028] In the following, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings in the embodiments of the present application, so that those skilled in the art can better understand the technical solutions of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first", "second", "target", "candidate", "alternative" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0030] Embodiment one

[0031] Figure 1 is a flowchart of the target detection method provided by the first embodiment of the present application; the present embodiment can be applied to the case of training a target detection distillation model with multiple detection heads to perform more effective and accurate product detection, which can be performed by a target detection device in the form of hardware and / or software. As shown in the figure, the target detection method comprises the following steps. Figure 1

[0032] S101, input the training image into the backbone network of the target detection network to extract features of the training image.

[0033] The training image can be an X-ray image, which is an image obtained by scanning and shooting an object with X-rays. The backbone network can include an input layer and an output layer, the input layer is used to input the training image, and the output layer is used to extract features of the training image in a convolutional manner based on a preset convolution algorithm.

[0034] ​Optionally, the target detection network comprises: a backbone network, a bottleneck layer, and a detection head; the backbone network is mainly used for feature extraction on an input image; the bottleneck layer is mainly used for feature fusion processing or coding and decoding processing on the extracted feature map, and the processed feature is transmitted to the detection head; the detection head performs final regression prediction to obtain a predicted target frame and a position and a category of the predicted target frame.

[0035] In S102, the features output by the backbone network are input into the first branch of the bottleneck layer and the first detection head for processing to obtain a first predicted target frame, and the features output by the backbone network are input into the second branch of the bottleneck layer and the second detection head for processing to obtain a second predicted target frame.

[0036] The first predicted target frame refers to a predicted target frame obtained by processing the features output by the backbone network in a first processing mode and target prediction, and the second predicted target frame refers to a predicted target frame obtained by processing the features output by the backbone network in a second processing mode and target prediction. The predicted target frame specifically refers to a final target frame corresponding to each target in a training image, a category and a position information corresponding to the target frame, and the like.

[0037] Optionally, the first branch can be a neural network with a coding and decoding structure, which is used for coding and decoding processing on the features output by the backbone network to make the determined background information and target information boundary of the training image more distinct; and the second branch can be a preset feature fusion algorithm, which is used for fusion on the features output by the backbone network to make the feature map have different receptive fields.

[0038] Optionally, when the features output by the backbone network are processed, the predicted target frame of each target in the training image can be determined based on the maximum matching principle of the Hungarian algorithm.

[0039] Optionally, the processing of the features output by the backbone network into the first branch of the bottleneck layer and the first detection head to obtain the first predicted target frame, and the processing of the features output by the backbone network into the second branch of the bottleneck layer and the second detection head to obtain the second predicted target frame, comprises: in the first branch of the bottleneck layer, processing the features output by the backbone network based on the encoder and the decoder of the Transformer network, and in the first detection head, performing target detection by using a feedforward network to obtain the first predicted target frame; in the second branch of the bottleneck layer, processing the features output by the backbone network based on a feature pyramid network, and in the second detection head, performing target detection by using a non-maximum suppression method to obtain the second predicted target frame.

[0040] The Transformer network is an attention mechanism-based graph neural network. The attention sublayer output in the encoding-decoding layer is connected to a feed-forward network (FFN), and the target detection is performed through the fully connected layer of the feed-forward network to obtain the first predicted target frame. The feature pyramid network (FPN) is a feature fusion network. The non maximum suppression (NMS) is an algorithm for target detection based on the removal of redundant detection frames.

[0041] Optionally, in the first branch of the bottleneck layer, the encoder and the decoder based on the Transformer network process the features output by the backbone network, and in the first detection head, the feed-forward network is used for target detection to obtain the first predicted target frame, including: based on the encoder, determining all receptive field features of the backbone network, performing encoding operation on all receptive field features, and performing self-attention operation on different receptive field features according to global attention sharing parameters to obtain feature maps with global attention; based on the decoder, decoding the feature maps output by the encoding layer to obtain decoding layer features and the correlation relationship between the decoding layer features; in the first detection head, based on the feed-forward network, target prediction is performed according to the decoding layer features and the correlation relationship between the decoding layer features to obtain the first predicted target frame.

[0042] The receptive field feature can represent the size of the region on the input training set picture that is mapped by the pixel point on the feature map output by the bottleneck layer, that is, each point on the receptive field feature map corresponds to a region on the input training image. The correlation relationship between the decoding layer features may, for example, be the correlation relationship between the features of different feature regions of an image, which can be represented by a correlation coefficient.

[0043] For example, the first processing manner for predicting the features output by the backbone network to obtain the first predicted target frame can be: the encoder and the decoder based on the Transformer network encode and decode the features output by the backbone network to obtain feature maps with global attention; and the feed-forward network is used to perform target detection using prior knowledge to obtain the prediction result of the target.

[0044] Optionally, based on the feature pyramid network, the features output by the backbone network are processed, and in the second detection head, a non-maximum suppression method is used for target detection to obtain a second predicted target frame, including: based on the feature pyramid network, the features output by the backbone network are fused to obtain feature maps with different receptive fields; the feature maps with different receptive fields are effectively extracted, and in the second detection head, a non-maximum suppression method is used for target detection to obtain a second predicted target frame.

[0045] It should be noted that after the output layer of the backbone network outputs the features, two different processing methods of the first branch plus the first detection head and the second branch plus the second detection head can be performed at the same time, that is, the bottleneck layer and the detection head are divided into two branches to determine different predicted target frames. The feature maps output by the first branch and the feature maps output by the second branch are the same in width, height and channel.

[0046] It should be noted that the process of target detection of the first branch and the second branch is as follows: the entire training image is divided into a plurality of detection frames, the category and position information of the object in each detection frame are predicted, and then the loss between the predicted frame and the real frame is used to screen the target frame, and finally the category and position information of the target frame are output, that is, the first predicted target frame and the second predicted target frame are obtained.

[0047] It should be noted that the target detection scheme proposed in the present application uses two different detection heads in a target detection network to participate in model training; and in the training stage, the multiple detection heads interact with each other, and the backbone network, the bottleneck layer and the detection head are guided to each other, which can effectively improve the target detection capability.

[0048] S103, based on the preset loss function, the size relationship between the first target loss value and the second target loss value is determined according to the first predicted target frame, the second predicted target frame and the real target frame, and the target detection model with smaller target loss value is determined as the teacher model, and the target detection model with larger target loss value is determined as the student model.

[0049] The preset loss function may be, for example, a regression loss function and a predicted target category loss function. The real target frame refers to the actual position region of each target in the training image known in advance. The target loss value refers to the loss value obtained by evaluating the prediction results obtained by using the first branch and the first detection head, and the second branch and the second detection head for prediction processing based on the preset loss function.

[0050] Optionally, after determining the first prediction target frame and the second prediction target frame by using two different processing methods, the same loss value calculation method can be used to calculate a first target loss value between the first prediction target frame and the real target frame, and a second target loss value between the second prediction target frame and the real target frame.

[0051] Specifically, based on the preset loss function, the size relationship between the first target loss value and the second target loss value is determined according to the first prediction target frame, the second prediction target frame and the real target frame, including:

[0052] (1) Based on the CIOU loss function, a first CIOU loss value between the first prediction target frame and the real target frame, and a second CIOU loss value between the second prediction target frame and the real target frame are determined. Wherein, the CIOU loss (Complete Intersection over Union Loss, Complete IoU Loss) is a regression positioning loss function considering three geometric parameters of overlapping area, center point distance and aspect ratio.

[0053] Exemplarily, the first CIOU loss value and the second CIOU loss value can be determined based on the following formula respectively:

[0054] bbox ciou score =CIOU (bbox pre, bbox gt)

[0055] Wherein, bbox ciou score is the first CIOU loss value between the first prediction target frame and the real target frame, or the second CIOU loss value between the second prediction target frame and the real target frame, bbox pre represents the coordinates of the prediction target frame, and bbox gt represents the coordinates of the real target frame.

[0056] (2) Based on the coordinate loss function, a first coordinate loss value between the first prediction target frame and the real target frame, and a second coordinate loss value between the second prediction target frame and the real target frame are determined. Exemplarily, the first coordinate loss value and the second coordinate loss value can be determined by the following formula.

[0057] Wherein, bbox pre represents the coordinates of the prediction target frame, and bbox gt represents the coordinates of the real target frame.

[0058]

[0059] (3) determining a first regression loss value between the first predicted target frame and the real target frame according to the first CIOU loss value, the first coordinate loss value, the CIOU weight corresponding to the CIOU loss and the coordinate weight corresponding to the coordinate loss; determining a second regression loss value between the second predicted target frame and the real target frame according to the second CIOU loss value, the second coordinate loss value, the CIOU weight corresponding to the CIOU loss and the coordinate weight corresponding to the coordinate loss;

[0060] For example, the first regression loss value and the second regression loss value can be determined based on the following formula:

[0061]

[0062] wherein, bbox ciou score is the CIOU loss value, is the coordinate loss value, λ ciou is the preset CIOU weight corresponding to the CIOU loss, λ2 is the preset coordinate weight corresponding to the coordinate loss, Loss bbox is the regression loss value.

[0063] (4) determining the size relationship between the first target loss value and the second target loss value according to the first regression loss value, the second regression loss value and the predicted target category loss value.

[0064] Optionally, the product of the classification loss (such as cross-entropy) and the CIOU loss value can be determined as the predicted target category loss value. Specifically, the predicted target category loss value Loss label can be determined based on the following formula:

[0065] Loss label = Loss cls × bbox ciou score

[0066] wherein, Loss cls is the loss value determined by using the existing classification loss calculation method (such as cross-entropy loss), bbox ciou score is the CIOU loss value between the predicted target frame and the real target frame.

[0067] Optionally, the first target loss value and the second target loss value are determined according to the first regression loss value, the second regression loss value and the predicted target category loss value, including: determining the sum of the first regression loss value and the predicted target category loss value as the first target loss value; determining the sum of the second regression loss value and the predicted target category loss value as the second target loss value. Further, the size relationship between the first target loss value and the second target loss value can be determined according to the first target loss value and the second target loss value.

[0068] Exemplarily, the target loss value Loss of the target detection model corresponding to the processing manner can be calculated based on the following formula sum :

[0069] Loss sum =Los label +Loss bbox

[0070] wherein, Los label is the prediction target category loss value, and Loss bbox is the regression loss value.

[0071] S104, based on the distillation manner, the teacher model and the student model are jointly trained to obtain the trained target detection model, and the trained target detection model is used to detect the target object in the to-be-detected image.

[0072] Exemplarily, the finally trained target detection model can be specifically used for detecting contraband in the to-be-detected image.

[0073] Optionally, based on the distillation manner, the teacher model and the student model are jointly trained to obtain the trained target detection model, comprising:

[0074] (1) respectively using the teacher model and the student model to predict the training image, and determining the feature maps corresponding to the teacher model and the student model;

[0075] (2) based on the preset calculation manner, the distance difference and the similarity difference between the two kinds of feature maps are determined, and according to the distance difference and the similarity difference, the first distillation loss in the bottleneck layer is determined, and the distillation operation in the bottleneck layer is realized;

[0076] wherein, the distance difference represents the difference of the feature values between the two kinds of feature maps.

[0077] Exemplarily, the distance difference dil_fpn_abs can be determined based on the following formula:

[0078] dil_fpn_abs=abs(feature_map_st,feature_map_te)

[0079] wherein, abs represents taking absolute value, feature_map_st is the feature map determined by using the student model for prediction processing. feature_map_te is the feature map determined by using the teacher model for prediction processing.

[0080] Exemplarily, the similarity difference dil_fpn_channel between the two feature maps can be determined based on the following formula:

[0081] dil_fpn_channel = abs(cov(feature_map_channel_st), cov(featur_map_channel_te))

[0082] wherein abs represents taking absolute value, cov represents performing covariance calculation to determine the correlation difference. feature_map_channel_st refers to the covariance of the feature map determined by the student model for prediction processing, and feature_map_channel_te refers to the covariance of the feature map determined by the teacher model for prediction processing.

[0083] Optionally, the sum of the distance difference and the similarity difference can be determined as the first distillation loss at the bottleneck layer, and the distillation operation at the bottleneck layer can be performed based on the first distillation loss.

[0084] (3) the prediction result of the teacher model is taken as the pseudo-label target of the student model for training, and the second distillation loss at the detection head is determined to realize the distillation operation at the detection head;

[0085] Exemplarily, the second distillation loss dil_loss at the detection head can be determined based on the following formula: dil_loss = KL_head kl + label_te score × bbox_ciou_te score (dil_bbox_head + dil_label_head) wherein KL_head kl is the category similarity of the target box predicted by the teacher model and the target box predicted by the student model, label_te score is the score of one category predicted by the teacher model. bbox_ciou_te score is the CIOU loss value obtained by training the teacher model. dil_bbox_head is the difference between the predicted box coordinates of the student model and the teacher model, taking the target predicted by the teacher model as the real target. dil_label_head is the difference between the predicted category of the student model and the predicted category of the teacher model, taking the target predicted by the teacher model as the real target.

[0086] It should be noted that since the student model is the true target predicted by the teacher model during distillation, but the teacher model will have errors, the application increases the confidence of the predicted target, and determines the second distillation loss dil_loss of the detection head by using the above formula, which can effectively avoid the above problems and improve the generalization performance of the finally generated target detection model.

[0087] Exemplarily, the score of one of the categories of the target predicted by the teacher model can be determined based on the following formula:

[0088]

[0089] wherein label sc ore represents the score of one of the categories of the predicted target, and x represents the feature value of the category of the frame.

[0090] Exemplarily, the category similarity KL_head between the target frame predicted by the teacher model and the target frame predicted by the student model can be determined based on the following formula: kl

[0091] KL_head kl =KL(P_teacher,P_student)

[0092] wherein KL() refers to calculating the KL divergence (Kullback-Leibler Divergence). P_teacher and P_student represent the category probability distribution of the target frame predicted by the teacher model and the student model, respectively.

[0093] Exemplarily, the difference dil_bbox_head between the target predicted by the teacher model and the target frame coordinate predicted by the student model can be determined based on the following formula:

[0094]

[0095] wherein bbox_te is the target frame coordinate determined by the teacher model, bbox_st is the target frame coordinate determined by the student model, and CIOU() refers to calculating the CIOU loss.

[0096] Exemplarily, the difference dil_label_head between the target predicted by the teacher model and the category predicted by the student model can be determined based on the following formula:

[0097] dil_label_head=cls(obj_te,obj_st)

[0098] ​Wherein, obj te and obj st represent the class of the target frame predicted by the teacher model and the student model respectively. cls() is the difference of the class.

[0099] (4) Determine the target distillation loss according to the first distillation loss and the second distillation loss, and perform distillation operation according to the target distillation loss to obtain the trained target detection model.

[0100] It should be noted that the target detection model distillation training scheme adopted by the present application can enable the teacher model and the student model to be trained together, reduce the training time, and the loss value of the teacher model and the student model is determined by the pre-network training, which is free from artificial intervention, so that the detection ability of the student model is not limited by the detection ability of the teacher model, and in addition, by adopting two different mechanism detection heads, distillation is performed at the bottleneck layer and the detection head respectively, the prediction frame and the class of the prediction target are predicted, which can effectively improve the accuracy of the target detection result.

[0101] The technical scheme of the embodiment of the present application inputs the training image into the backbone network of the target detection network to perform feature extraction on the training image; inputs the features output by the backbone network into the first branch of the bottleneck layer and the first detection head for processing to obtain the first prediction target frame; and inputs the features output by the backbone network into the second branch of the bottleneck layer and the second detection head for processing to obtain the second prediction target frame; based on a preset loss function, the size relationship between the first target loss value and the second target loss value is determined according to the first prediction target frame, the second prediction target frame and the real target frame, and the target detection model with the smaller target loss value is determined as the teacher model, and the target detection model with the larger target loss value is determined as the student model; based on the distillation mode, the teacher model and the student model are trained together to obtain the trained target detection model, and the trained target detection model is used to detect dangerous goods in the product according to the X-ray image of the product after X-ray irradiation. In this way, different detection heads can be added for model training, making the distillation training process more intelligent, so as to obtain a target detection model with higher accuracy for detecting target objects in the image to be detected.

[0102] Embodiment two

[0103] Figure 2 is a flowchart of the target detection method provided by the second embodiment of the present application; the present embodiment gives a preferred example of using two different detection heads to perform target detection processing (i.e. using FFN and NMS methods for target detection in the detection head) for target detection based on the above-mentioned embodiment.

[0104] As Figure 2 shown, the target detection method can include the following processes:

[0105] After the training image input target detection network is processed by the backbone network, the processing can be predicted from two branches, specifically, the first branch is processed by the encoder and decoder of the Transformer network at the bottleneck layer, and further processed by the FFN at the detection head for target detection; the second branch is processed by the FPN for feature fusion at the bottleneck layer, and further processed by the NMS method at the detection head for target detection; after the two branches are determined as the teacher model and the student model according to the loss value obtained by training, the first distillation operation (denoted as distillation 1) can be performed at the bottleneck layer, the second distillation operation (denoted as distillation 2) can be performed at the detection head, and the distillation operation can be performed according to the finally determined target distillation loss, so as to obtain the trained target detection model, so as to facilitate subsequent target detection of the input target object, such as detecting the target object in the to-be-detected image.

[0106] Specifically, the present application adopts two different detection heads for distillation. Specifically, the features output by the backbone network are processed based on two branches, and two kinds of processing are performed respectively:

[0107] ①The output features are encoded and decoded to obtain feature maps with global attention, and the full connection layer of the feedforward network is used to predict the target;

[0108] ②The output layer of the backbone network adopts convolution for feature extraction, and further based on the feature pyramid network and the non-maximum suppression method, the prior knowledge is used to predict the prediction box to obtain the prediction target.

[0109] Further, the output layer of the backbone network and the detection head are distilled respectively, and the specific implementation method is as follows:

[0110] Step one: training image data processing, input to the backbone network for feature extraction;

[0111] Step two: the features extracted by the backbone network are simultaneously processed by two operations;

[0112] Operation one:

[0113] ①The backbone network is input to the encoding branch, and the encoding layer is performed first:

[0114] It should be noted that the encoding layer of the present application performs global attention operation on different receptive fields of the backbone network. Since the second branch is used for processing in actual application, encoding operation is performed on all receptive field feature layers, which only increases a small amount of time for training, but saves time compared with the traditional distillation method. Since all receptive field feature layers are encoded, the model has good detection rate for small targets and global attention sharing parameters for different receptive field feature maps, thereby accelerating the learning of the encoding layer.

[0115] The decoding layer decodes the feature map output by the encoding layer, and determines the relationship between the feature value obtained by the encoding layer and other feature layers, so that the boundary between background information and target information is more distinct.

[0116] The feature value of the decoding layer performs target prediction through the FFN.

[0117] The Hungarian algorithm is used to obtain the predicted target, and loss calculation is performed with the real target to obtain a loss value.

[0118] Operation two:

[0119] The features of the output of the backbone network are fused to make the feature map have different receptive fields, and then effective feature extraction is performed to strengthen the image information and semantic information of the feature map. However, due to the limitations of convolution, the fused feature map does not have the global information of operation one, but the calculation amount is sharply reduced during reasoning.

[0120] The feature map after feature fusion performs target prediction.

[0121] The predicted target is subjected to NMS operation, and loss calculation is performed with the real target to obtain a loss value.

[0122] Optionally, operation one and operation two share a set of loss calculation methods. Since the traditional target detection network model calculates the target score and the classification score separately in the training stage, but combines the calculation of the target score and the classification score in the test stage, the calculation methods in the training stage and the test stage are inconsistent, which leads to the absence of correlation between the target box score and the classification score distribution. In addition, since the CIOU loss calculation focuses on the proportion of the overlapping area of the predicted target and the real target, the attention to the coordinates is lacking. Therefore, the present application increases the coordinate loss calculation after the CIOU loss calculation, so that the target box loss has both area proportion loss and coordinate loss, thereby solving the proportion of the predicted target and the real target and the influence of the target box being too large or too small. The specific target loss value calculation method is described in detail in embodiment one S103, and will not be repeated here.

[0123] It should be noted that the encoding and decoding structure of the first branch keeps the width, height and channel of the input and output consistent, and the size of the feature layer of the second branch keeps the feature layer consistent. Therefore, the feature map output by the first branch and the feature map output by the second branch have the same width, height and channel. According to the target loss value, the target loss value smaller is determined as the teacher model, and the target loss value larger is determined as the student model.

[0124] It should be noted that, in the early training stage, the weights in the encoding and decoding structure are uniformly assigned, which causes deviation in the effective feature extraction capability, and the guidance through the second branch can accelerate the effective feature extraction capability, and in the later training stage, the encoding and decoding structure has a self-attention extraction capability, and the global context extraction capability of the second branch is improved, so that the bottleneck layer distillation can reduce the training period and improve the detection capability of the target detection model.

[0125] The technical scheme of the present application adopts two or more different types of detection heads to perform self-distillation on the target detection model, and performs distillation operations at the bottleneck layer and the detection head respectively, achieving distillation of the same network; the student model and the teacher model are judged by the loss value, and the decision is made by the model, reducing human intervention, and the teacher model and the student model are trained together, reducing the training time and solving the problem of limited detection capability of the student model in the past; the convolutional detection network and the Transformer detection network are effectively combined, the traditional convolutional detection network is used to accelerate the training period of the Transformer detection network, and the Transformer detection network is used to increase the global context information of the traditional convolutional detection network without increasing the inference time; when calculating the regression loss of the target detection model, the area ratio and the target coordinate difference are effectively combined, so that the regression loss is not affected by the size and position difference of the target frame.

[0126] Embodiment three

[0127] Figure 3 is the structure block diagram of the target detection device provided by the embodiment three of the present application; the embodiment can be applicable to the case of training a target detection distillation model using multiple detection heads for more effective and accurate product detection, and the target detection device can be realized in the form of hardware and / or software and configured in a device with target detection function, such as Figure 3 As shown in the figure, the device specifically includes:

[0128] The extraction module 301 is configured to input the training image into the backbone network of the target detection network to extract features of the training image;

[0129] The obtaining module 302 is configured to input the features output by the backbone network into the first branch and the first detection head of the bottleneck layer for processing to obtain the first predicted target frame; and input the features output by the backbone network into the second branch and the second detection head of the bottleneck layer for processing to obtain the second predicted target frame;

[0130] The determination module 303 is configured to determine the size relationship between the first target loss value and the second target loss value based on the preset loss function according to the first predicted target frame, the second predicted target frame and the real target frame, and determine the target detection model with the smaller target loss value as the teacher model and the target detection model with the larger target loss value as the student model.

[0131] The detection module 304 is configured to jointly train the teacher model and the student model based on a distillation manner to obtain a trained target detection model, and detect a target object in a to-be-detected image by using the trained target detection model.

[0132] The technical scheme of the embodiment of the present application inputs a training image into a backbone network of a target detection network to perform feature extraction on the training image; inputs the features output by the backbone network into a first branch of a bottleneck layer and a first detection head for processing to obtain a first predicted target frame; and inputs the features output by the backbone network into a second branch of the bottleneck layer and a second detection head for processing to obtain a second predicted target frame; determines the size relationship between a first target loss value and a second target loss value based on a preset loss function, the first predicted target frame, the second predicted target frame and a real target frame, and determines a target detection model with a smaller target loss value as a teacher model and a target detection model with a larger target loss value as a student model; and jointly trains the teacher model and the student model based on a distillation manner to obtain a trained target detection model, and detects a target object in a to-be-detected image by using the trained target detection model. In this way, different detection heads and bottleneck layer branches can be added for model training, the distillation training process is more intelligent, and a target detection model with higher accuracy can be obtained for detecting a target object in a to-be-detected image.

[0133] Further, the target detection network comprises a backbone network, a bottleneck layer and a detection head; the backbone network is mainly used for feature extraction on an input image; the bottleneck layer is mainly used for feature fusion processing or coding and decoding processing on the extracted feature map, and the processed features are transmitted to the detection head; the detection head performs regression prediction to obtain a predicted target frame and the position and category of the predicted target frame.

[0134] Further, the obtaining module 302 can comprise:

[0135] The first detection unit is configured to process the features output by the backbone network based on the encoder and the decoder of the Transformer network in the first branch of the bottleneck layer, and perform target detection by using the feedforward network in the first detection head to obtain a first predicted target frame.

[0136] The second detection unit is configured to process the features output by the backbone network based on the feature pyramid network in the second branch of the bottleneck layer, and perform target detection by using the non-maximum suppression method in the second detection head to obtain a second predicted target frame.

[0137] Further, the first detection unit is specifically configured to:

[0138] Based on the encoder, all the receptive field features of the backbone network are determined, encoding operations are performed on all the receptive field features, and self-attention operations are performed on different receptive field features according to global attention shared parameters to obtain feature maps with global attention.

[0139] Based on the decoder, the feature maps output by the encoding layer are decoded to obtain decoding layer features and correlation relationships between the decoding layer features.

[0140] In the first detection head, target prediction is performed based on the feedforward network according to the decoding layer features and the correlation relationships between the decoding layer features to obtain the first predicted target frame.

[0141] Further, the second detection unit is specifically configured to:

[0142] Based on the feature pyramid network, the features output by the backbone network are fused to obtain feature maps with different receptive fields.

[0143] The effective feature extraction is performed on the feature maps with different receptive fields, and target detection is performed based on the non-maximum suppression method in the second detection head to obtain the second predicted target frame.

[0144] Further, the determination module 303 is specifically configured to:

[0145] Based on the CIOU loss function, the first CIOU loss value between the first predicted target frame and the real target frame and the second CIOU loss value between the second predicted target frame and the real target frame are determined.

[0146] Based on the coordinate loss function, the first coordinate loss value between the first predicted target frame and the real target frame and the second coordinate loss value between the second predicted target frame and the real target frame are determined.

[0147] According to the first CIOU loss value, the first coordinate loss value, the CIOU weight corresponding to the CIOU loss, and the coordinate weight corresponding to the coordinate loss, the first regression loss value between the first predicted target frame and the real target frame is determined.

[0148] According to the second CIOU loss value, the second coordinate loss value, the CIOU weight corresponding to the CIOU loss, and the coordinate weight corresponding to the coordinate loss, the second regression loss value between the second predicted target frame and the real target frame is determined.

[0149] According to the first regression loss value, the second regression loss value, and the predicted target category loss value, the size relationship between the first target loss value and the second target loss value is determined.

[0150] Further, the detection module 304 is specifically configured to:

[0151] The teacher model and the student model are respectively used to predict the training image, and the feature maps corresponding to the teacher model and the student model are determined;

[0152] Based on the preset calculation mode, the distance difference and the similarity difference between the two feature maps are determined, and the first distillation loss at the bottleneck layer is determined according to the distance difference and the similarity difference, so as to realize the distillation operation at the bottleneck layer;

[0153] The prediction result of the teacher model is used as the pseudo-label target of the student model for training, and the second distillation loss at the detection head is determined, so as to realize the distillation operation at the detection head;

[0154] According to the first distillation loss and the second distillation loss, the target distillation loss is determined, and the distillation operation is performed according to the target distillation loss, so as to obtain the trained target detection model.

[0155] Embodiment four

[0156] Figure 4 is a structural schematic diagram of an electronic device provided in Embodiment Four of the present application. Figure 4 A structural schematic diagram of an electronic device 10 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0157] As shown in Figure 4 The electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11, wherein the memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0158] A plurality of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0159] The processor 11 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the object detection method.

[0160] In some embodiments, the object detection method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded onto the RAM 13 and executed by the processor 11, one or more steps of the object detection method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the object detection method by any other appropriate means, such as by means of firmware.

[0161] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0162] Computer programs for implementing the methods of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer program, when executed, enables the functions / acts specified in the flowcharts and / or block diagrams to be implemented. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package and partially on a remote machine or entirely on a remote machine or server.

[0163] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0164] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0165] The systems and techniques described herein can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described herein, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0166] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0167] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be executed in parallel, executed in series, or executed in different orders, as long as the desired results of the technical solutions of the present disclosure can be achieved, and the present disclosure is not limited herein.

[0168] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A target detection method characterized by, The method comprises the steps of: inputting a training image into a backbone network of a target detection network to perform feature extraction on the training image; inputting the features output by the backbone network into a first branch of a bottleneck layer and a first detection head to perform processing, so as to obtain a first predicted target frame; inputting the features output by the backbone network into a second branch of the bottleneck layer and a second detection head to perform processing, so as to obtain a second predicted target frame; based on a preset loss function, determining the size relationship between a first target loss value and a second target loss value according to the first predicted target frame, the second predicted target frame and a real target frame, and determining a target detection model with a smaller target loss value as a teacher model and a target detection model with a larger target loss value as a student model; based on a distillation mode, jointly training the teacher model and the student model to obtain a trained target detection model, and using the trained target detection model to detect a target object in a to-be-detected image; inputting a training image into a backbone network of a target detection network to perform feature extraction on the training image; inputting the features output by the backbone network into a first branch of a bottleneck layer and a first detection head to perform processing, so as to obtain a first predicted target frame; and inputting the features output by the backbone network into a second branch of the bottleneck layer and a second detection head to perform processing, so as to obtain a second predicted target frame, comprising: in the first branch of the bottleneck layer, processing the features output by the backbone network based on an encoder and a decoder of a Transformer network, and in the first detection head, performing target detection using a feedforward network to obtain the first predicted target frame; 2. The method of claim 1, wherein, in the second branch of the bottleneck layer, processing the features output by the backbone network based on a feature pyramid network, and in the second detection head, performing target detection using a non-maximum suppression method to obtain the second predicted target frame. processing the features output by the backbone network based on an encoder and a decoder of a Transformer network, and in the first detection head, performing target detection using a feedforward network to obtain the first predicted target frame, comprising: based on the encoder, determining all receptive field features of the backbone network, performing encoding operation on all receptive field features, and performing self-attention operation on different receptive field features based on global attention sharing parameters to obtain feature maps with global attention; based on the decoder, decoding the feature maps output by the encoding layer to obtain decoding layer features and the correlation relationship between the decoding layer features; 3. The method of claim 1, wherein, in the first detection head, based on the feedforward network, target prediction is performed according to the decoding layer features and the correlation relationship between the decoding layer features to obtain the first predicted target frame. processing the features output by the backbone network based on a feature pyramid network, and in the second detection head, performing target detection using a non-maximum suppression method to obtain the second predicted target frame, comprising: based on the feature pyramid network, performing feature fusion on the features output by the backbone network to obtain feature maps with different receptive fields; performing effective feature extraction on the feature maps with different receptive fields, and in the second detection head, performing target detection based on the non-maximum suppression method to obtain the second predicted target frame.

4. The method of claim 1, wherein, Determine the size relationship between the first target loss value and the second target loss value according to the first predicted target frame, the second predicted target frame and the real target frame based on the preset loss function, including: Determine the first CIOU loss value between the first predicted target frame and the real target frame, and the second CIOU loss value between the second predicted target frame and the real target frame based on the CIOU loss function; Determine the first coordinate loss value between the first predicted target frame and the real target frame, and the second coordinate loss value between the second predicted target frame and the real target frame based on the coordinate loss function; Determine the first regression loss value between the first predicted target frame and the real target frame according to the first CIOU loss value, the first coordinate loss value, the CIOU weight corresponding to the CIOU loss and the coordinate weight corresponding to the coordinate loss; Determine the second regression loss value between the second predicted target frame and the real target frame according to the second CIOU loss value, the second coordinate loss value, the CIOU weight corresponding to the CIOU loss and the coordinate weight corresponding to the coordinate loss; Determine the size relationship between the first target loss value and the second target loss value according to the first regression loss value, the second regression loss value and the predicted target category loss value.

5. The method of claim 1, wherein, Based on the distillation mode, the teacher model and the student model are trained together to obtain a trained target detection model, including: Respectively adopt the teacher model and the student model to predict and process the training image, and determine the feature maps corresponding to the teacher model and the student model; Based on the preset calculation mode, determine the distance difference and the similarity difference between the two kinds of feature maps, and according to the distance difference and the similarity difference, determine the first distillation loss in the bottleneck layer to realize the distillation operation in the bottleneck layer; The prediction result of the teacher model is used as the pseudo label target of the student model for training, and the second distillation loss in the detection head is determined to realize the distillation operation in the detection head; According to the first distillation loss and the second distillation loss, determine the target distillation loss, and perform distillation operation according to the target distillation loss to obtain a trained target detection model.

6. A target detection apparatus characterized by comprising: Including: The extraction module is used for inputting the training image into the backbone network of the target detection network to extract features of the training image; The obtaining module inputs the features output by the backbone network into the first branch and the first detection head of the bottleneck layer for processing to obtain the first predicted target frame; And input the features output by the backbone network into the second branch and the second detection head of the bottleneck layer for processing to obtain the second predicted target frame; The determination module is used for determining the size relationship between the first target loss value and the second target loss value according to the first predicted target frame, the second predicted target frame and the real target frame based on the preset loss function, and determining the target detection model with smaller target loss value as the teacher model and the target detection model with larger target loss value as the student model; The detection module is used for training the teacher model and the student model together based on the distillation mode to obtain a trained target detection model, and detecting the target object in the to-be-detected image by using the trained target detection model; The obtaining module includes: The first detection unit is configured to, in a first branch of the bottleneck layer, process the features output by the backbone network based on an encoder and a decoder of a Transformer network, and in a first detection head, perform target detection on the processed features by using a feedforward network to obtain first predicted target boxes. The second detection unit is configured to, in a second branch of the bottleneck layer, process the features output by the backbone network based on a feature pyramid network, and in a second detection head, perform target detection on the processed features by using a non-maximum suppression method to obtain second predicted target boxes.

7. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the target detection method in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and the computer instructions are used to enable the processor to implement the target detection method in any one of claims 1-5 when executed.

Images