A decoupled knowledge distillation harness target detection method and system

Abstract

The present application relates to a kind of decoupling knowledge distillation hardware target detection method and system, method includes: according to data set mark frame in first model and second model respectively, decoupling is carried out to mark data set, and the foreground target area and background area are obtained, to determine the target kind probability of hardware, and according to target kind probability determine common feature score mask;According to common feature score mask and feature map classification score graph determine first distillation loss function;Based on global context module, according to the determination feature map, construct second distillation loss function;According to each distillation loss function, determine total loss function, and according to total loss function, the second model is trained, to carry out target detection to the hardware image to be measured.The present application uses the common shape feature of different kinds of hardware, adopts decoupling knowledge distillation method, for foreground target area, the common feature belonging to the unique hardware between is used for the information integration of target classification to migrate to small model, improves hardware detection precision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection and identification, in particular to a decoupled knowledge distillation hardware target detection method and system. BACKGROUND

[0002] Hardware is a widely used metal accessory in power transmission lines, used to support, fix, connect conductors, conductors and insulators, etc. Hardware on the power transmission line is prone to damage, rust, deformation and other faults due to long-term outdoor environment, which has an adverse effect on the safe and stable transmission of electricity. Accurate detection of hardware targets is the basis for hardware defect detection, and is of great significance for convenient inspection of power systems.

[0003] Most of the current inventions for hardware target detection are to build complex and deep networks to improve detection accuracy. This increases the number of model parameters and requires high computing power of the computing device, which is not conducive to deploying to front-end devices for fast and accurate hardware target detection. Lightweight deployment of high-performance models to mobile detection devices for power system inspection has become a research hotspot. SUMMARY

[0004] In order to overcome the shortcomings of the prior art, the purpose of the present application is to provide a decoupled knowledge distillation hardware target detection method and system.

[0005] To achieve the above purpose, the present application provides the following solutions:

[0006] A decoupled knowledge distillation hardware target detection method, comprising:

[0007] Decouple each image in the labeled data set according to the data set label frame in the first model and the second model, respectively, to obtain a first foreground target region, a first background region, a second foreground target region and a second background region; the labeled data set includes multiple labeled hardware aerial images; the parameter quantity and performance of the first model are greater than those of the second model;

[0008] Determine the target type probability of the hardware according to the first foreground target region, and determine the common feature score mask according to the target type probability;

[0009] Determine a first distillation loss function according to the common feature score mask, the first model and the feature map classification score map of the feature map pyramid network and a preset convolution function;

[0010] Based on the global context module, a second distillation loss function is constructed according to the first feature map determined by the first foreground target region and the first background region, and the second feature map determined by the second foreground target region and the second background region;

[0011] determine a total loss function according to the first distillation loss function and the second distillation loss function, and train the second model according to the total loss function to obtain a trained second model; the trained second model is used for target detection on a to-be-tested hardware fitting image.

[0012] Preferably, the acquired annotation dataset is decoupled according to the first model and the second model respectively, so as to obtain a first foreground target region, a first background region, a second foreground target region and a second background region, and the method comprises the following steps of:

[0013] constructing an annotation dataset; the annotation dataset comprises the hardware fitting aerial image with a ground-truth box and annotation information;

[0014] based on the annotation dataset, decoupling the hardware fitting target from the image with the ground-truth box as a boundary;

[0015] designing a binary mask, and controlling whether a feature map belongs to a foreground target region or a background region according to the binary mask.

[0016] Preferably, a formula of the binary mask is as follows:

[0017] wherein, M (i,j) is the binary mask, i is a horizontal direction coordinate of a feature map generated by a feature pyramid network of the first model or the second model, j is a vertical direction coordinate of the feature map generated by the feature pyramid network of the first model or the second model, and G is the ground-truth box.

[0018] Preferably, a target category probability of the hardware fitting is determined according to the first foreground target region, and a common feature score mask is determined according to the target category probability, and the method comprises the following steps of:

[0019] based on the first foreground target region, randomly sampling features in a feature map with horizontal and vertical directions being i and j respectively;

[0020] determining the target category probability according to a first formula; the first formula is as follows: Y S = M (i,j) P(S|f); wherein, S represents a class number of a target category of the hardware fitting, f is the feature, Y S represents a probability that the feature f is the target category S, M (i,j) uses the binary mask;

[0021] modeling the target category probability by using a deep neural network to obtain a second formula; the second formula is as follows: Y S = M (i,j)P(S|f, θ); wherein θ represents a model of generation parameters;

[0022] Based on the second formula, aggregate the target category probability of all categories to obtain a common feature score; the formula of the common feature score is Wherein F' is the common feature score;

[0023] According to the common feature score, determine a common feature score mask; the formula of the common feature score mask is: Wherein F is the common feature score mask; s' represents all accessory objects S as a sample, s' ∈ [1, s].

[0024] Preferably, according to the common feature score mask, the first model and the second model, determine a first distillation loss function in a feature map pyramid network feature map classification score map and a preset convolution function, comprising:

[0025] According to the common feature score mask, determine a score standard mask; the formula of the score standard mask is: Wherein F T is the score standard mask, Y s' T is the classification score map of the first model, θ T is the classification result of the accessory of the first model classification head, f T is the feature input to the first model classification head;

[0026] Use the aggregate classification score map of each layer of the feature pyramid network as a common feature score mask; the formula of the common feature score mask is: Wherein F k is the common feature score mask of the first model on channel C in the kth layer feature map pyramid network, Y kC T is the classification score map of the first model in the kth layer, and the channel number is C feature pyramid network;

[0027] According to the preset convolution function, the common feature score mask and the score standard mask, determine the first distillation loss function; the formula of the first distillation loss function is: Wherein L front is the first distillation loss function, n represents the total number of layers of the feature map pyramid network, k represents the kth layer of the feature map pyramid, (i, j) represents the position of the feature map, W represents the width, H represents the height, l represents the total number of channels, C represents the corresponding channel, is the common feature score mask at position (i, j), N kThe sum of the feature map common feature score masks for all feature maps, f adap The function is the preset convolution function that adapts the first model and the second model channel, F T kC(i,j) And F S kC(i,j) Respectively, the first model and the second model in the feature map pyramid network corresponding layer, corresponding channel, the same position feature map classification score map.

[0028] Preferably, the formula of the second distillation loss function is:

[0029] L rela = μ∑(R(F T )-R(F S )) 2 ;

[0030] Wherein, L rela is the second distillation loss function, μ is the hyperparameter of balancing loss function, F T , F S Respectively, the feature map generated by the first model and the second model, R is the relationship information of the target and the background area captured by different models for a certain target, and the formula of R is: Wherein, W v1 , W v2 , W k Respectively, the first linear transformation matrix, the second linear transformation matrix and the third linear transformation matrix, F i is the feature map of the input instance, Is the weight of global attention pooling, LN represents normalization processing, N p is the number of feature layers, N p = H·W, e is a natural constant, F M is the generated feature map for the first foreground target area; F j is any extracted feature in the image, δ(·) = W v2 Relu(LN(W v1) (·)) represents a feature transformation that captures channel dependency.

[0031] Preferably, the formula of the total loss function is:

[0032] L = αL front + βL rela ;

[0033] Wherein, L is the total loss function, and α and β are different balancing parameters.

[0034] A decoupled knowledge distillation target detection system, comprising:

[0035] A decoupling module is configured to decouple the labeled data set according to data set label frames in a first model and a second model respectively, so as to obtain a first foreground target region, a first background region, a second foreground target region and a second background region correspondingly; the labeled data set comprises a plurality of labeled aerial images of fittings; the parameter quantity and performance of the first model are greater than those of the second model;

[0036] A scoring module is configured to determine a target category probability of the fitting according to the first foreground target region and the first background region, and determine a common feature score mask according to the target category probability;

[0037] A first distillation module is configured to determine a first distillation loss function according to the common feature score mask, the first model and the second model, a feature map classification score map of a feature map pyramid network and a preset convolution function;

[0038] A second distillation module is configured to construct a second distillation loss function according to a first feature map determined according to the first foreground target region and the first background region and a second feature map determined according to the second foreground target region and the second background region based on a global context module;

[0039] A total loss function determination module is configured to determine a total loss function according to the first distillation loss function and the second distillation loss function, and train the second model according to the total loss function to obtain a trained second model; the trained second model is used for target detection on a to-be-tested fitting image.

[0040] According to the specific embodiments of the present application, the following technical effects are provided:

[0041] The application provides a decoupling knowledge distillation goldware target detection method and system, the method comprising: decoupling a labeled data set according to data set bounding boxes in a first model and a second model respectively, to obtain a first foreground target region, a first background region, a second foreground target region and a second background region; the labeled data set comprises multiple labeled goldware aerial images; the parameter quantity and performance of the first model are greater than those of the second model; a target category probability of the goldware is determined according to the first foreground target region and the first background region, and a common feature score mask is determined according to the target category probability; a first distillation loss function is determined according to the common feature score mask, a feature map classification score map of a feature map pyramid network and a preset convolution function; a second distillation loss function is constructed based on a global context module, according to the first feature map determined according to the first foreground target region and the first background region, and the second feature map determined according to the second foreground target region and the second background region; a total loss function is determined according to the first distillation loss function and the second distillation loss function, and the second model is trained according to the total loss function to obtain a trained second model; the trained second model is used for target detection of a to-be-tested goldware image. The application makes full use of the common shape features of different types of goldware, adopts a decoupling knowledge distillation method, and integrates the common features unique to the goldware into the information of target classification for migration to a small model, thereby improving the problem of decreased detection accuracy of the lightweight network for the goldware in a complex background aerial image. The application is practical, has achieved good accuracy improvement effect, shortens the required training time, does not require human participation, and has certain reference significance for the design of solutions to related problems. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0043] Figure 1 The flowchart of the decoupling knowledge distillation goldware target detection method in the embodiments provided by the present application;

[0044] Figure 2 The overall training block diagram in the embodiments provided by the present application;

[0045] Figure 3 The data set schematic diagram according to the standard labeling in the embodiments provided by the present application;

[0046] Figure 4A decoupling process schematic diagram in the embodiment provided by the present application;

[0047] Figure 5 A foreground target area shared feature score schematic diagram in the embodiment provided by the present application;

[0048] Figure 6 A small model individual training result schematic diagram in the embodiment provided by the present application;

[0049] Figure 7 A large model individual training result schematic diagram in the embodiment provided by the present application;

[0050] Figure 8 A large model using the distillation method to guide small model training result schematic diagram in the embodiment provided by the present application. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0052] In this document, reference to“an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase“in an embodiment” in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive of one another. Those skilled in the art will appreciate that embodiments described herein can be combined with other embodiments.

[0053] The terms“first”,“second”,“third”, and“fourth” and the like in the description and in the claims, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. Furthermore, the terms“comprise”,“include”,“contain” and“have” and their conjugates, as used in the specification and in the claims, shall not be construed as limiting the application to the specific elements or steps described. For example, the expression“comprising a step” shall not be construed as excluding the presence of additional steps beyond those described.

[0054] The purpose of the present application is to provide a decoupling knowledge distillation fitting target detection method and system, which can significantly improve the fitting detection accuracy.

[0055] In order to make the above-mentioned purposes, characteristics and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0056] The method described in the embodiment is based on the knowledge distillation technology, which is one of the effective algorithms for lightweight networks. A model with large parameter quantity and superior performance is referred to as a large model (i.e., the first model in the embodiment), and a model with small parameter quantity and poor performance is referred to as a small model (i.e., the second model in the embodiment). The knowledge distillation technology can improve the performance of the small model by transferring the beneficial knowledge in the large model to the small model. Since the gold accessory aerial image contains a large amount of complex background, the gold accessory target accounts for a small proportion in the complete aerial image, and the proportion is different in different images. The feature information of the foreground target and the background in the image is not differentiated from the large model to the small model, and the performance of the small model cannot be improved.

[0057] Figure 1 The flowchart of the decoupled knowledge distillation gold accessory target detection method in the embodiment provided by the present application is shown in Figure 1 The present application provides a decoupled knowledge distillation gold accessory target detection method, which comprises the following steps:

[0058] Step 100: Decouple the annotated data set according to the data set annotation box in the first model and the second model, respectively, to obtain the first foreground target region, the first background region, the second foreground target region and the second background region; the annotated data set comprises a plurality of annotated gold accessory aerial images; the parameter quantity and performance of the first model are greater than those of the second model;

[0059] Step 200: Determine the target category probability of the gold accessory according to the first foreground target region, and determine the common feature score mask according to the target category probability;

[0060] Step 300: Determine the first distillation loss function according to the common feature score mask, the first model and the second model in the feature map pyramid network, the feature map classification score map and the preset convolution function;

[0061] Step 400: Based on the global context module, the second distillation loss function is constructed according to the first feature map determined by the first foreground target region and the first background region, and the second feature map determined by the second foreground target region and the second background region;

[0062] Step 500: Determine the total loss function according to the first distillation loss function and the second distillation loss function, and train the second model according to the total loss function to obtain the trained second model; the trained second model is used for target detection of the to-be-tested gold accessory image.

[0063] Figure 2 The overall training block diagram in the embodiment provided by the present application is shown in Figure 2 The specific steps are as follows:

[0064] Step 1, according to the standard construction of the fitting data set, so that the image with ground-truth box and label information storage to xml file (such as Figure 3 is a data set example, where the boundary box in the figure is a manually annotated ground-truth box, and the area contained in the boundary box is the fitting target foreground area, and the other area is the background information area).

[0065] Step 2, decouple the fitting target with the ground-truth box as the boundary of the whole image, and the decoupled fitting foreground target area and background information area (such as Figure 4 shown), Figure 4 the right upper corner is the decoupled foreground target area, the area in each ground-truth box is used to aggregate the common features between fittings, and the unique feature information of the fitting is migrated to the small model to improve the classification accuracy of target detection, and the lower right corner is the background information area. Decoupling by area to avoid migrating redundant background information.

[0066] Step 3, for the foreground target area, the large model and the small model score the random features at each layer of the feature pyramid network, and the classification probability of the common features of the fitting by the large model is used as the scoring standard in the distillation process, that is, the classification score map of the features by the large model (such as Figure 5 shown, in the foreground target area, the large model and the small model score the random features, and the feature information used by the large model for classification of the common features of the fitting is used to guide the model. This is the main distillation method adopted for the foreground target area), which extracts classification feature knowledge for guiding small model training.

[0067] Step 4, the fitting installation position is special, and the transmission line feature information in the background area is one of the favorable hints for fitting detection. The position information and appearance feature information between fittings have a significant prompting effect on detection accuracy. In order to strengthen the association between fitting targets and background information, GcBlock module is adopted to capture the relationship between a fitting target and other fitting targets and the background environment, which is conducive to the large model using complete information to guide the training of the small model to improve the precision of the small model. In the semantic context modeling part, 1x1 convolution and softmax function are used to obtain attention weights, and then matrix multiplication operation is used to make the attention weights affect the original input size CxHxW feature map, so as to obtain global features. This operation is to reduce the parameter amount in calculation.

[0068] Step 5, first train the large network, and let the trained large network guide the training of the small network with this distillation method to improve performance visualization as shown in Figure 6 , 7 , 8. Figure 6The test results of the small model trained and tested alone without guidance show that the small model has poor positioning and classification ability for fittings, Figure 7 The test results of the large model show that the large model has good detection ability. Figure 8 The test results of the small model guided by the large model using the method of the application show that the detection performance of the small model has been significantly improved and even exceeds the performance of the large model. The names of the detected targets in the figure are: shockproof hammer, bag-type suspension, hanging board, adjusting board, and weight.

[0069] Preferably, the step 100 specifically comprises:

[0070] Constructing a labeled data set, wherein the labeled data set includes the aerial images of the fittings with ground-truth boxes and labeled information;

[0071] Based on the labeled data set, decoupling the fittings targets from the background based on the ground-truth boxes;

[0072] Designing a binary mask and controlling whether the feature map belongs to the foreground target region or the background region according to the binary mask.

[0073] Specifically, the embodiment first decouples and processes the fittings targets and the background:

[0074] During inspection, the shooting angle of the unmanned aerial vehicle is variable, and the power transmission line is in a variable and complex environment. For safety, the unmanned aerial vehicle shooting should have a certain safety distance, resulting in small fittings targets and large background in the shooting images. In order to reduce the adverse effects of excessive background information on target positioning and classification, the fittings targets and the background decoupling region are distilled. The ground-truth box is used as the boundary to decouple into a foreground target region and a background region, a binary mask M is designed to control whether the feature map belongs to the foreground target region or the background information region, and (i, j) represents the horizontal and vertical coordinates of the feature map generated by the feature pyramid network of the model. If (i, j) is within the ground-truth boundary box, it is determined that the feature belongs to the foreground target region, and the mask M (i,j) = 1; if (i, j) is not within the ground-truth boundary box, it is determined that the feature map is a background region, and M (i,j) = 0, wherein G represents the ground-truth box, as shown in formula (1):

[0075]

[0076] Preferably, the binary mask is formulated as:

[0077] wherein M (i,j) is the binary mask, i is the coordinate in the horizontal direction of the feature map generated by the feature pyramid network of the first model or the second model, j is the coordinate in the vertical direction of the feature map generated by the feature pyramid network of the first model or the second model, and G is the ground-truth box.

[0078] Preferably, the step 200 specifically comprises:

[0079] Based on the first foreground target region, randomly sampling features within the feature map in the horizontal and vertical directions, respectively, i and j;

[0080] According to a first formula, determining the target category probability; the first formula is: Y S = M (i,j) P(S|f); wherein S represents the number of categories of the target category of the fitting, and f is the feature; Y S represents the probability that the feature f is the target category S, M (i,j) uses the binary mask;

[0081] Using a deep neural network to model the target category probability, obtaining a second formula; the second formula is: Y S = M (i,j) P(S|f, θ); wherein θ represents a model that generates parameters;

[0082] Based on the second formula, aggregating the target category probabilities of all categories to obtain a common feature score; the formula of the common feature score is wherein F' is the common feature score.

[0083] According to the common feature score, determining a common feature score mask; the formula of the common feature score mask is: wherein F is the common feature score mask; s' represents all fitting objects S as a sample, and s' ∈ [1, s].

[0084] The embodiment secondly distills the foreground target region according to the common feature score:

[0085] Randomly sampling features f within the feature map in the horizontal and vertical directions, respectively, i and j, and the possibility that these features are a certain fitting target can be described as shown in formula (2):

[0086] Y S = M (i,j) P(S|f) (2)

[0087] S represents a target category, Y S P (S | f, θ) represents the probability that the feature f is the target category S, M (i,j) is used to control whether it is a foreground target area. Such conditional probability can be modeled by a deep neural network, and θ represents the model generated as a parameter as shown in equation (3):

[0088] Y S = M (i,j) P (S | f, θ) (3)

[0089] The proposed common feature score concept is defined as the probability that these features belong to a certain gold target. In order to aggregate all categories of gold categories, this method aggregates the probabilities of all categories to obtain the common feature score, and here the maximum value is used to aggregate, as shown in equation (4).

[0090]

[0091] Since there are many kinds of gold objects, the present application represents all categories of gold in the data set with s, and let s' ∈ [1, s], s' as a sample to represent all gold objects S. Therefore, the common feature score of all gold categories is represented by F as a mask, as shown in equation (5):

[0092]

[0093] In the distillation process, the classification probability Y s' T of the large model on the gold is taken as the common feature score standard, that is, the classification score map of the large model on the feature, and it is considered that the classification result of the large model has been trained. The classification standard for guiding the small model training. At the same time, the classification result θ T of the large model classification head on the gold is taken as the specific generation parameter model instance of θ, f T is the feature input to the large model classification head. In this way, the large model's score standard for the feature is used to guide the small model to extract features in the form of mask F T , as shown in equation (6).

[0094]

[0095] The feature pyramid network is proposed to deal with the multi-scale target detection problem. The shallow network feature map has high resolution and is more suitable for detecting small scale targets, and the deep network feature map has low resolution and is more suitable for detecting large scale targets. In order to completely use the classification knowledge of large network to guide the small model, a common feature score mask should be generated for each layer of the feature pyramid network to represent the probability of objects of different scales and different resolutions. The aggregation classification score map of each layer of the feature pyramid network is used as the common feature score mask, as shown in formula (7):

[0096]

[0097] F k is the common feature score mask of the kth layer of the feature pyramid network of the large model on channel C, Y kC T is the classification score map of the large model at the kth layer of the feature pyramid network with channel number C. Therefore, the distillation loss of the small model guided by the common feature score mask of the large model in the feature pyramid network in the foreground target area is as shown in formula (8):

[0098]

[0099] Where n represents the total number of layers of the feature pyramid network, k represents the kth layer of the feature pyramid, (i, j) represents the position of the feature map, W represents the width, H represents the height, l represents the total number of channels, and C represents the corresponding channel. F k(i,j) is the common feature score mask at position (i, j), f adap is a convolution function that adapts the channels of the large model and the small model, F T kC(i,j) and F S kC(i,j) are the classification score maps of the large model and the small model at the corresponding layer of the feature pyramid network, the corresponding channel, and the same position feature map, respectively. The loss function is used to make the small model learn the classification ability of the large model.

[0100] Preferably, the step 300 specifically comprises:

[0101] A score standard mask is determined according to the common feature score mask; the formula of the score standard mask is: Where F T is the score standard mask, Y s' T is the classification score map of the first model, θ T is the classification result of the first model classification head for the fitting, and f Tfeatures input to the first model classification head;

[0102] an aggregated classification score map of each layer of the feature pyramid network is used as a shared feature score mask; the formula of the shared feature score mask is: wherein F k is a shared feature score mask of the feature pyramid network of the kth layer of the first model on channel C, Y kC T is a classification score map of the feature pyramid network of the first model on channel C at the kth layer;

[0103] the first distillation loss function is determined according to the preset convolution function, the shared feature score mask and the score standard mask; the formula of the first distillation loss function is: wherein L front is the first distillation loss function, n represents the total number of layers of the feature pyramid network, k represents the kth layer of the feature pyramid, (i,j) represents the position of the feature map, W represents the width, H represents the height, l represents the total number of channels, C represents the corresponding channel, is the shared feature score mask at the position (i,j), N k is the sum of the shared feature score masks of all feature maps, f adap is the preset convolution function for adapting the channels of the first model and the second model, F T kC(i,j) and F S kC(i,j) are classification score maps of the first model and the second model respectively at the corresponding layer of the feature pyramid network, the corresponding channel and the same position of the feature map.

[0104] Preferably, the formula of the second distillation loss function is:

[0105] L rela = μ∑(R(F T )-R(F S )) 2 ;

[0106] wherein L rela is the second distillation loss function, μ is a hyperparameter for balancing the loss function, F T , F S are feature maps generated by the first model and the second model respectively, R is relationship information of a target and a background region captured by different models for a target, and the formula of R is: wherein W v1 , W v2 , W krespectively, are a first linear transformation matrix, a second linear transformation matrix and a third linear transformation matrix, F i is a feature map of an input instance, is a weight of global attention pooling, LN represents a normalization process, N p is a position number of a feature layer, N p = H·W, e is a natural constant, F M is a generated feature map of the first foreground target region; F j is an arbitrary extracted feature in an image, δ(·) = W v2 Relu(LN(W v1) (·)) represents a feature transformation that captures channel dependency.

[0107] Preferably, a formula of the total loss function is:

[0108] L = aL front + bL rela ;

[0109] wherein L is the total loss function, a is a first coefficient, and b is a second coefficient.

[0110] The embodiment finally strengthens the connection of the whole image and the relationship between the target and the background:

[0111] The embodiment focuses on the most important region of the hardware target detection. However, the hardware installation position on the power transmission line is fixed, and the surrounding environmental information plays a significant role in improving the understanding of the hardware installation position. There is an important relationship between different pixels and targets. If only the feature knowledge of the foreground target region is transferred, the relationship between the hardware target and the background is broken. The knowledge guided by the large model is not complete enough, which makes the small model lose the perception ability of the whole image. The guiding method of only transferring the foreground target feature knowledge does not fully transfer the whole image knowledge to the small model. The small model cannot learn the systematic and complete knowledge, so the performance cannot reach the optimal. The GcBlock (Global Context Block) global semantic modeling module is used to capture the relationship between the image hardware and the hardware, and between the hardware and the background information. The small model is prompted to learn the perception ability of the relationship information of the large model. The representation of the whole image relationship distillation loss is shown in formula (10):

[0112] L rela = m∑(R(F T )-R(F S )) 2 (9)

[0113] wherein m is a hyperparameter for balancing the loss function, F T , F SThe feature maps generated by the large model and the small model are respectively. The R function is the relationship information between the target and the background area captured by different models for a certain target, as shown in formula (11):

[0114]

[0115] Among them W v1 W v2 W k F represents different linear transformation matrices. i The feature map of an input instance These are the weights for global attention pooling, where LN represents normalization, and N... p N represents the number of locations in the feature layer; for an image, N represents the number of locations in the feature layer. p =H·W, where e is the natural constant, F M It is a feature map of the generated foreground region; F j It is the arbitrary extraction of features from the image, δ(·)=W v2 Relu(LN(W v1) (·) represents a feature transformation that captures channel dependencies. The GcBlock module includes global attention pooling operations for context modeling and bottleneck transformations to capture channel correlations.

[0116] Therefore, the decoupled knowledge distillation target detection method based on shared feature scoring uses the total loss function of formula (12) to perform knowledge distillation on the small model:

[0117] L=αL front +βL rela (11)

[0118] The large model only performs knowledge distillation on the feature map of the small model, so it can be applied to a variety of target detection networks, making various models for metal object detection lightweight.

[0119] Corresponding to the above method, this embodiment also provides a decoupled knowledge distillation fitting target detection system, including:

[0120] The decoupling module is used to decouple the labeled dataset from the first model and the second model respectively based on the labeled bounding boxes in the dataset, thereby obtaining a first foreground target region, a first background region, a second foreground target region, and a second background region; the labeled dataset includes multiple labeled aerial images of hardware; the first model has a larger number of parameters and higher performance than the second model;

[0121] The scoring module is used to determine the target type probability of the hardware based on the first foreground target region and the first background region, and to determine the common feature scoring mask based on the target type probability.

[0122] a first distillation module configured to determine a first distillation loss function according to the shared feature score mask, the first model, the second model, a feature map classification score map of a feature map pyramid network, and a preset convolution function;

[0123] a second distillation module configured to determine a second distillation loss function according to the first feature map determined based on the first foreground target region and the first background region and the second feature map determined based on the second foreground target region and the second background region based on a global context module;

[0124] a total loss function determination module configured to determine a total loss function according to the first distillation loss function and the second distillation loss function, and train the second model according to the total loss function to obtain a trained second model, wherein the trained second model is configured to perform target detection on a to-be-tested fitting image.

[0125] The present application has the following advantages:

[0126] The present application first decouples the fitting foreground target region and the background region according to the ground-truth (real effective value) frame labeled in the labeled data set; for the foreground target region, the feature information of each layer of the feature map pyramid network is classified and scored, and the classification scores of all categories are aggregated as a shared feature score mask, and the feature information represented by the mask is the main migration knowledge of the large model guiding the small model; pure knowledge distillation in the foreground target region breaks the integrity of the image, and the relationship module is adopted for the background information to capture the relationship between the target and other targets and the background to obtain prompt information conducive to target detection, so as to achieve the purpose of migrating the complete information of each image and improve the detection accuracy of the small model. The present application makes full use of the shared shape features of different types of fittings, adopts a decoupling knowledge distillation method, integrates the unique shared features between fittings belonging to the foreground target region into the small model for target classification information migration, and improves the detection accuracy of the lightweight network for fittings in complex background aerial images. The present application is practical, has achieved good precision improvement effect, shortens the training time, does not require human participation, and has certain reference significance for the design of related problems.

[0127] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0128] The principles and implementation manners of the present application are described by using specific examples in the present application, and the above examples are only used to help understand the method of the present application and its core idea; meanwhile, for the general technical personnel in the art, the specific implementation manners and application ranges will be changed according to the idea of the present application. In conclusion, the content of the present specification should not be understood as the limitation of the present application.

Claims

1. A method for decoupling knowledge distillation fittings target detection, characterized in that, include: The labeled dataset is decoupled from the first model and the second model based on the labeled bounding boxes in the dataset, respectively, to obtain the first foreground target region, the first background region, the second foreground target region, and the second background region; the labeled dataset includes multiple labeled aerial images of hardware. The first model has a larger number of parameters and higher performance than the second model; The target type probability of the hardware is determined based on the first foreground target region and the first background region, and a common feature scoring mask is determined based on the target type probability. The first distillation loss function is determined based on the shared feature scoring mask, the feature map classification score map of the first model and the second model in the feature map pyramid network, and the preset convolution function. Based on the global context module, a second distillation loss function is constructed according to the first feature map determined by the first foreground target region and the first background region, and the second feature map determined by the second foreground target region and the second background region; The total loss function is determined based on the first distillation loss function and the second distillation loss function, and the second model is trained based on the total loss function to obtain the trained second model; the trained second model is used to perform target detection on the image of the metal fittings to be tested.

2. The decoupled knowledge distillation fitting target detection method according to claim 1, characterized in that, The step of decoupling the obtained labeled dataset according to the first model and the second model respectively, to obtain the first foreground target region, the first background region, the second foreground target region and the second background region, includes: Construct a labeled dataset; the labeled dataset includes aerial images of the hardware with ground-truth bounding boxes and labeled information; Based on the labeled dataset, the hardware target is decoupled from the image using the ground-truth box as the boundary; Design a binary mask, and control whether the feature map belongs to the foreground target region or the background region based on the binary mask.

3. The decoupled knowledge distillation fitting target detection method according to claim 2, characterized in that, The formula for the binary mask is: Among them, M (i,j) Let i be the horizontal coordinate of the feature map generated by the feature pyramid network of the first or second model, j be the vertical coordinate of the feature map generated by the feature pyramid network of the first or second model, and G be the ground-truth box.

4. The decoupled knowledge distillation fitting target detection method according to claim 3, characterized in that, The target type probability of the hardware is determined based on the first foreground target region and the first background region, and a common feature scoring mask is determined based on the target type probability, including: Based on the first foreground target region, features are randomly selected from the feature maps i and j in the horizontal and vertical directions, respectively. The probability of the target type is determined according to the first formula; the first formula is: Y S =M (i,j) P(S|f); where S represents the number of target types of hardware, and f is the feature; Y S M represents the probability that the feature f is the target class S. (i,j) Use the binary mask described above; The probability of the target type is modeled using a deep neural network, resulting in a second formula; the second formula is Y. S =M (i,j) P(S|f,θ); where θ represents the model for generating parameters; Based on the second formula, the probabilities of the target categories for all classes are aggregated to obtain a common feature score; the formula for the common feature score is as follows: Wherein, F′ is the shared feature score; A common feature score mask is determined based on the common feature scores; the formula for the common feature score mask is: Where F is the shared feature scoring mask; s' is used as a sample to represent all hardware objects S, s'∈[1,s].

5. The decoupled knowledge distillation fitting target detection method according to claim 4, characterized in that, The step of determining the first distillation loss function based on the shared feature scoring mask, the feature map classification score maps of the first and second models in the feature map pyramid network, and a preset convolution function includes: The scoring criterion mask is determined based on the shared feature scoring mask; the formula for the scoring criterion mask is: Among them, F T Y is the mask for the scoring criteria. s' T For the classification score map of the first model, θ T f represents the classification result of the first model's classification head for hardware. T Features are input to the classification head of the first model; The aggregated classification score map of each layer of the feature pyramid network is used as a common feature score mask; the formula for the common feature score mask is: Among them, F k Y is the common feature scoring mask of the k-th layer feature map pyramid network of the first model on channel C. kC T The first model is shown as a classification score map of a feature pyramid network with C channels at layer k. The first distillation loss function is determined based on the preset convolution function, the common feature scoring mask, and the scoring criterion mask; the formula for the first distillation loss function is: Among them, L front Let be the first distillation loss function, n represent the total number of layers in the feature map pyramid network, k represent the k-th layer of the feature map pyramid, (i,j) represent the position of the feature map, the width is W, the height is H, l represents the total number of channels, and C represents the corresponding channel. F k(i,j) For the shared feature scoring mask at position (i,j), N k f is the sum of the common feature score masks for all feature maps. adap The function is the preset convolution function that adapts the channels of the first model and the second model. T kC(i,j) and F S kC(i,j) The first model and the second model are respectively the feature map classification scores of the first model and the second model at the corresponding layer, corresponding channel, and the same position in the feature map pyramid network.

6. The decoupled knowledge distillation fitting target detection method according to claim 5, characterized in that, The formula for the second distillation loss function is: L rela =μ∑(R(F T )-R(F S )) 2 ; Among them, L rela Let F be the second distillation loss function, μ be the hyperparameter of the equilibrium loss function, and F be the value of F. T F S The first and second models generate feature maps respectively, and R represents the relationship information between the hardware target and the background area captured by different models for a certain hardware target. The formula for R is: Among them W v1 W v2 W k These are the first linear transformation matrix, the second linear transformation matrix, and the third linear transformation matrix, respectively, F. i For the feature map of the input instance, These are the weights for global attention pooling, where LN represents normalization, and N... p N represents the number of locations in the feature layer. p =H·W, where e is the natural constant, F M This is the feature map generated for the first foreground target region; F j To extract any feature from the image, δ(·)=W v2 Relu(LN(W v1) (·)) represents the feature transformation that captures channel dependencies.

7. The method for decoupling knowledge distillation fittings target detection according to claim 6, characterized in that, The formula for the total loss function is: L=αL front +βL rela ; Where L is the total loss function, α is the first coefficient, and β is the second coefficient.

8. A decoupled knowledge distillation fitting target detection system, characterized in that, include: The decoupling module is used to decouple the labeled dataset from the first model and the second model respectively based on the labeled bounding boxes in the dataset, thereby obtaining the first foreground target region, the first background region, the second foreground target region, and the second background region; the labeled dataset includes multiple labeled aerial images of hardware. The first model has a larger number of parameters and higher performance than the second model; The scoring module is used to determine the target type probability of the hardware based on the first foreground target region and the first background region, and to determine the common feature scoring mask based on the target type probability. The first distillation module is used to determine the first distillation loss function based on the shared feature scoring mask, the feature map classification score map of the first model and the second model in the feature map pyramid network, and a preset convolution function. The second distillation module is used to construct a second distillation loss function based on the global context module, according to the first feature map determined by the first foreground target region and the first background region, and the second feature map determined by the second foreground target region and the second background region. The total loss function determination module is used to determine the total loss function based on the first distillation loss function and the second distillation loss function, and to train the second model based on the total loss function to obtain a trained second model; the trained second model is used to perform target detection on the image of the metal fittings to be tested.

Images