A fine-grained feature adversarial alignment method and system for cross-domain object detection

Abstract

The application discloses a fine-grained feature adversarial alignment method and system for cross-domain target detection, comprising the following steps: step 1, according to the positioning result of a target detection model, extracting the target boundary box inner features of a source domain and a target domain from the positioning result; step 2, according to the classification result of the target detection model, first obtaining the class labels corresponding to the target boundary box inner features of the source domain and the target domain, and then combining the class labels with coarse-grained domain labels to obtain fine-grained domain labels; step 3, according to the target boundary box inner features of the source domain and the target domain, obtaining the fine-grained domain classification scores of the target boundary box inner features of the source domain and the target domain through a fine-grained domain classifier; and step 4, according to the fine-grained domain labels and the fine-grained domain classification scores, calculating a fine-grained feature adversarial alignment loss.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, and particularly to a fine-grained feature adversarial alignment method and system for cross-domain target detection. BACKGROUND

[0002] When a trained target detection model is deployed to a brand new environment, it often falls into the dilemma of detection performance degradation caused by domain shift. In recent years, adversarial learning domain-invariant feature representation has become an effective strategy to alleviate the above problems. Among them, the target detection model plays a role similar to the generator in the generative adversarial network (GAN), and the domain classifier plays the role of the discriminator. The gradient reversal layer (GRL) located between the two is equivalent to an identity mapping during the network forward propagation process, and the gradient sign is negated during the network back propagation process. When trained to a certain extent, the domain discriminator cannot distinguish whether the features extracted by the target detection model come from the source domain or the target domain, thereby achieving preliminary alignment of source domain features and target domain features. SUMMARY

[0003] The present application aims to provide a fine-grained feature adversarial alignment method and system for cross-domain target detection to overcome or at least alleviate at least one of the above-mentioned deficiencies of the prior art.

[0004] To achieve the above-mentioned purpose, the present application provides a fine-grained feature adversarial alignment method for cross-domain target detection, which comprises:

[0005] Step 1: According to the positioning result of the target detection model, the source domain and target domain target bounding box inside features are extracted from the positioning result;

[0006] Step 2: According to the classification result of the target detection model, first, the class labels corresponding to the source domain and target domain target bounding box inside features are obtained, and then the class labels and coarse-grained domain labels are combined to obtain fine-grained domain labels;

[0007] Step 3: According to the source domain and target domain target bounding box inside features, the fine-grained domain classification scores of the source domain and target domain target bounding box inside features are obtained through the fine-grained domain classifier;

[0008] Step 4: According to the fine-grained domain labels and the fine-grained domain classification scores, the fine-grained feature adversarial alignment loss is calculated.

[0009] Further, the method of "combining the class labels and coarse-grained domain labels into fine-grained domain labels" in step 2 specifically comprises:

[0010] Set the coarse-grained domain label corresponding to the source domain data to [1, 0], and set the coarse-grained domain label corresponding to the target domain data to [0, 1].

[0011] The coarse-grained domain label vector [1, 0] and the 0 elements in the vector [0, 1] are transformed into category labels c of the features within the i-th target bounding box at dimension K. i Same zero vector;

[0012] The category label c of the features within the i-th target bounding box i Decomposed according to its dimension K into Then, concatenate the zero vectors of the same dimension to obtain the fine-grained domain labels of the features within the bounding box of the i-th source domain target. Fine-grained domain labels of features within the bounding box of the i-th target domain

[0013] in, The dimension of the category labels representing the features within the target bounding box is K. The fine-grained domain classification scores representing features within the target bounding box all have a dimension of 2K.

[0014] Furthermore, step 2, "obtaining the category labels corresponding to the features within the target bounding boxes of the source and target domains," is specifically achieved through one of the following two methods:

[0015] The first approach involves using formula (1) to perform softmax processing on the class classification scores of the features within the i-th target bounding box, obtaining the class probability scores:

[0016]

[0017] In the formula, p i，j y i，j y i，k p represents the class probability score of the feature within the bounding box of the i-th target. i The j-th element, category classification score y i The j-th element and the k-th element, where j = 1, 2, ..., K;

[0018] Then, element p i，j The class label c of the feature within the i-th target bounding box that corresponds to the maximum value i The j-th element c i，j Set it to 1, and set the category label c i Set other elements to 0 to obtain the hard label of the category of the feature within the bounding box of the i-th target;

[0019] The second way, first, the temperature coefficient T adjusts the smoothing degree of the softmax distribution of the class classification score of the feature in the i-th target bounding box, and obtains the class probability score p

[0020]

[0021] As shown in equation (4), the class probability score p i as a class soft label:

[0022] c i，j = p i，j (4).

[0023] Further, step 3 specifically includes:

[0024] Step 31, respectively expand each neuron of the ordinary domain classifier output layer to K neurons, and the constructed fine-grained domain classifier includes: M2 full connection layers, and the first (M2-1) full connection layers are followed by an activation function, and the last full connection layer is followed by a softmax function, and the number of neurons in the i-th full connection layer of the fine-grained domain classifier is represented by N 2，i , where i = 1, 2, …, M2;

[0025] Step 32, taking the feature in the i-th source domain or target domain target bounding box as the input of the fine-grained domain classifier, the fine-grained domain classifier predicts that the fine-grained domain classification score of the i-th source domain target bounding box feature is The fine-grained domain classification score of the i-th target domain target bounding box feature is Wherein, represents the j'th element of the fine-grained domain classification score of the i-th source domain target bounding box feature, represents the j'th element of the fine-grained domain classification score of the i-th target domain target bounding box feature, j' = 1, 2, …, 2K.

[0026] Further, step 4 specifically includes:

[0027] The j'th element of the fine-grained domain probability score of the i-th source domain target bounding box feature is obtained by performing softmax processing on the fine-grained domain classification score of the i-th source domain target bounding box feature through equation (5), and the j'th element of the fine-grained domain probability score of the i-th target domain target bounding box feature is obtained by performing softmax processing on the fine-grained domain classification score of the i-th target domain target bounding box feature through equation (6)

[0028]

[0029] ​

[0030] The fine-grained feature adversarial alignment loss of the features in the source domain target bounding box is calculated by formula (7) The fine-grained feature adversarial alignment loss of the features in the target domain target bounding box is calculated by formula (8)

[0031]

[0032]

[0033] Further, the fine-grained feature adversarial alignment method for cross-domain target detection further comprises:

[0034] Step 5, the fine-grained feature adversarial alignment loss of the features in the source domain and target domain target bounding box Merging into the loss of the target detection model itself, training the target detection model combined with the fine-grained domain classifier, and obtaining the optimal model weight for adaptive cross-domain target detection.

[0035] Further, step 5 specifically comprises:

[0036] By formula (9), the sum of the fine-grained feature adversarial alignment loss of the features in the source domain and target domain target bounding box is adjusted by using the coefficient α, and then added to the target detection model loss Therefore, the total loss of the target detection model combined with the fine-grained domain classifier is obtained

[0037]

[0038] Taking formula (9) as the optimization target, the target detection model combined with the fine-grained domain classifier is trained, and when After convergence is completed, the optimal model weight in the current training process is selected.

[0039] The application also provides a fine-grained feature adversarial alignment system for cross-domain target detection, which comprises:

[0040] A source domain and target domain target bounding box feature extraction module is configured to extract the features in the target bounding box of the source domain and the target domain from the positioning result according to the positioning result of the target detection model;

[0041] A source domain and target domain fine-grained domain label synthesis module is configured to obtain the class labels corresponding to the features in the target bounding box of the source domain and the target domain according to the classification result of the target detection model, and then combine the class labels and the coarse-grained domain labels into fine-grained domain labels;

[0042] a source domain and a target domain fine-grained domain classification score acquisition module configured to obtain fine-grained domain classification scores of the source domain and the target domain target bounding box features by a fine-grained domain classifier according to the source domain and the target domain target bounding box features;

[0043] a fine-grained feature adversarial alignment loss calculation module configured to calculate a fine-grained feature adversarial alignment loss according to the fine-grained domain labels and the fine-grained domain classification scores.

[0044] Further, the fine-grained feature adversarial alignment loss calculation module is further configured to adjust the sum of the fine-grained feature adversarial alignment losses of the source domain and the target domain target bounding box features by a coefficient α in formula (9), and then add the adjusted sum to the known target detection model loss to obtain a total loss of the target detection model combined with the fine-grained domain classifier Take formula (9) as an optimization target to train the target detection model combined with the fine-grained domain classifier, and when After convergence is completed, the optimal model weight in the current training process is selected;

[0045]

[0046] In the formula, is the fine-grained feature adversarial alignment loss of the source domain target bounding box features calculated by formula (7), is the fine-grained feature adversarial alignment loss of the target domain target bounding box features calculated by formula (8):

[0047]

[0048]

[0049] In the formula, is the j'th element in the fine-grained domain probability score of the i'th source domain target bounding box feature, is the j'th element in the fine-grained domain probability score of the i'th target domain target bounding box feature, is the j'th element in the fine-grained domain label of the i'th source domain target bounding box feature, is the j'th element in the fine-grained domain label of the i'th target domain target bounding box feature, n s , n t are respectively the total number of source domain and target domain local feature maps, K is the dimension of the class label of the source domain and the target domain target bounding box features, and j'=1, 2, …, 2K.

[0050] The present application has the following advantages due to the above technical solutions:

[0051] The application fully learns the domain-invariant feature representation through adversarial training, and then assists the target detection model to reasonably align the source domain and target domain fine-grained features in the training process, thereby fundamentally solving the problem that the existing ordinary domain classifier can only align the overall features of the source domain and the target domain. The method and system alleviate the classification difficulty of the target detection model for different categories of target instances, help to improve the classification accuracy of the target detection model after locating the target instance, and make the target detection model achieve further performance improvement in the cross-domain promotion process. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 The architecture diagram of the fine-grained feature adversarial alignment method for cross-domain target detection provided by the embodiment of the application.

[0053] Figure 2 The flowchart of the fine-grained feature adversarial alignment method for cross-domain target detection provided by the embodiment of the application.

[0054] Figure 3 The principle diagram of the ordinary domain classifier provided by the embodiment of the application.

[0055] Figure 4 The principle diagram of the fine-grained domain classifier in the Figure 3

[0056] Figure 5 The architecture diagram of the fine-grained feature adversarial alignment system for cross-domain target detection provided by the embodiment of the application. DETAILED DESCRIPTION

[0057] In the drawings, the same or similar notations are used to indicate the same or similar elements or elements having the same or similar functions. The embodiments of the application will be described in detail below with reference to the drawings.

[0058] In the case of no conflict, the technical features in each embodiment and each implementation form of the application can be combined with each other, and are not limited to the embodiment or implementation form in which the technical features are located.

[0059] The application will be further described below with reference to the drawings and specific embodiments. It should be pointed out that only an optimized technical solution is used to describe the technical solution and design principle of the application in detail below, but the protection scope of the application is not limited to this.

[0060] This document relates to the following terms, and for the sake of understanding, the meanings are as follows. Those skilled in the art should understand that the following terms can also have other names, but any other name should be considered consistent with the terms listed herein without departing from their meanings.

[0061] As Figure 1 and​Figure 2 As shown, the method for fine-grained feature alignment for cross-domain target detection provided by the embodiments of the present application comprises:

[0062] Step 1. According to the positioning result of the target detection model, the source domain target bounding box inside feature and the target domain target bounding box inside feature are extracted from the positioning result.

[0063] In one embodiment, step 1 specifically comprises:

[0064] Step 11. According to the positioning result of the target detection model, the source domain local feature map and the target domain local feature map are obtained. The embodiments of the present application are based on the unsupervised domain adaptive target detection scenario, at this time, the source domain data is labeled (bounding box label and category label), the target domain data is completely unlabeled, and the joint distribution of the target domain data is different from that of the source domain data. Wherein, the source domain data and the target domain data refer to the most original data input into the neural network.

[0065] No matter whether the source domain data or the target domain data is input into the target detection model, the target detection model will finally predict the positioning result, i.e. several sets of bounding box regression parameters of the target, on the feature map corresponding to the input data at a certain scale. The bounding box defined by each set of regression parameters is used to intercept the feature map at this scale to obtain the source domain local feature map and the target domain local feature map.

[0066] In one embodiment, the positioning result comprises ns sets of bounding box regression parameters predicted by the target detection model for the source domain data and nt sets of bounding box regression parameters predicted by the target detection model for the target domain data. That is, the total number of source domain local feature maps is n s , and the total number of target domain local feature maps is n t . Wherein, s represents the source domain, and t represents the target domain. As a person skilled in the art should understand, the target detection model can be an existing deep learning-based 2D or 3D target detection model (without cross-domain target detection capability), or an existing deep learning-based domain adaptive 2D or 3D target detection model (with limited cross-domain target detection capability).

[0067] Step 12. Feature extraction is performed on the source domain and target domain local feature maps to obtain the source domain target bounding box inside feature and the target domain target bounding box inside feature.

[0068] Specifically, the source domain or target domain local feature map obtained in step 11 is first passed through a pooling layer, then passed through M1 combinations of a fully connected layer and an activation layer, and finally the corresponding target bounding box feature is extracted. The pooling layer uses a pooling technology such as RoIAlign and RoI Pooling that can convert local feature maps of different heights and widths into local feature maps of the same height and width. The number of neurons of the fully connected layer is N1, and the activation layer can use ReLU, Leaky ReLU, or other forms of activation functions. M1 can be set according to actual conditions, and the default is 2. N1 takes an exponential power of 2, such as 512, 1024, 2048, or 4096.

[0069] In step 2, according to the classification result of the target detection model, the class labels corresponding to the source domain target bounding box features and the class labels corresponding to the target domain target bounding box features are obtained, and then the class labels and the coarse-grained domain labels are combined to obtain fine-grained domain labels.

[0070] In one embodiment, step 2 specifically includes:

[0071] In step 21, the class labels corresponding to the source domain target bounding box features and the class labels corresponding to the target domain target bounding box features are obtained through the classification result of the target detection model.

[0072] The target detection model not only predicts the position of the bounding box containing the target, but also predicts the class to which the target in the bounding box belongs. Because only the source domain data has labels and the target domain data has no labels, the source domain data is used to supervise the target detection model, and the positioning and classification capabilities of the model itself are improved through the training of the target detection model. The class classification scores of the source domain target bounding box features and the target domain target bounding box features are converted to obtain the class labels of the source domain target bounding box features and the class labels of the target domain target bounding box features.

[0073] In one embodiment, the set of class labels of the source domain target bounding box features is denoted as The set of class labels of the target domain target bounding box features is denoted as Wherein, denote the class label of the i-th source domain target bounding box feature and the class label of the i-th target domain target bounding box feature, respectively, denote the class label of the i-th source domain target bounding box feature and the class label of the i-th target domain target bounding box feature, respectively, and the upper subscript s of the source domain and the upper subscript t of the target domain in and are omitted, and c i represents the class label of the i-th source domain or target domain target bounding box feature.

[0074] The "obtaining the class labels corresponding to the features in the source domain and the target domain within the target bounding boxes" in step 2 can be implemented by one of the following methods:

[0075] The first method specifically includes:

[0076] In step 21a, the class classification score of the i-th feature within the target bounding box is subjected to softmax processing by formula (1) to obtain the class probability score. Whether it is a feature within the target bounding box in the source domain or a feature within the target bounding box in the target domain, the target detection model will finally output the corresponding classification result, i.e., the class classification score in the text. Here, the class classification score directly predicted by the target detection model for the i-th feature within the target bounding box is denoted as wherein, The dimension of the class classification score of the i-th feature within the target bounding box is K.

[0077]

[0078] In the formula, p i，j represents the j-th element of the class probability score p i of the i-th feature within the target bounding box, i.e., the probability that the i-th feature within the target bounding box belongs to the j-th class, y i，j represents the j-th element of the class classification score y i of the i-th feature within the target bounding box, y i，k represents the k-th element of the class classification score y i of the i-th feature within the target bounding box, and j = 1, 2, …, K.

[0079] In step 22a, the j-th element c i，j of the class label c i of the i-th feature within the target bounding box corresponding to the maximum element p i，j is set to 1, while the other elements of the class label c i are set to 0, thereby obtaining the class hard label of the i-th feature within the target bounding box. This process can be described by formula (2).

[0080]

[0081] The second method specifically includes:

[0082] In step 21b, the temperature coefficient T described by formula (3) is used to adjust the smoothing degree of the softmax distribution of the class classification score of the i-th feature within the target bounding box to obtain the class probability score. The class classification score y iThe adjusted class classification score is subjected to softmax processing by dividing by the temperature coefficient T to obtain the adjusted class probability score p of the i-th target bounding box feature i .

[0083]

[0084] In the formula, the temperature coefficient T is used to adjust the smoothness of the softmax distribution. When T = 1, formula (3) is equivalent to formula (1); when T < 1, the softmax distribution becomes sharper as T decreases; when T > 1, the softmax distribution becomes smoother as T increases.

[0085] Step 22b, as shown in formula (4), the class probability score p of the target bounding box feature i As a class soft label, therefore, the j-th element of the class soft label c of the i-th target bounding box feature i is:

[0086] c i，j = p i，j (4).

[0087] Of course, other existing methods can also be used to implement the step 2 of "obtaining the class labels corresponding to the source domain and target domain target bounding box features".

[0088] In one embodiment, the specific method of "coarse-grained domain label corresponding to the source domain and target domain target bounding box features" in step 2 includes:

[0089] The general domain classifier will only make a binary classification prediction on the input source domain data and target domain data. If the first element of the coarse-grained domain label used to supervise the general domain classifier is used as the source domain label and the second element is used as the target domain label, then the coarse-grained domain label corresponding to the source domain data is [1, 0], and the coarse-grained domain label corresponding to the target domain data is [0, 1].

[0090] The above coarse-grained domain label is a coarse-grained domain label, and after the general domain classifier is trained in an adversarial manner with the target detection model, it can only align the source domain and target domain features from the overall level, but cannot align the source domain and target domain fine-grained features from the class level.

[0091] In order to facilitate the supervision of the subsequently constructed fine-grained domain classifier during the training process of the target detection model, it is necessary to pre-integrate the class information into the coarse-grained domain label to synthesize the fine-grained domain label. Therefore, the method of "merging the class label and the coarse-grained domain label into the fine-grained domain label" in step 2 uses the splicing operation of two vectors in the same dimension direction, which specifically includes:

[0092] The coarse-grained domain label corresponding to the source domain data is set as [1, 0], and the coarse-grained domain label corresponding to the target domain data is set as [0, 1].

[0093] Whether it is a category hard label or a category soft label, the category label of the i-th target bounding box feature can be decomposed into Referring to the 0 element in the coarse-grained domain label vector [1, 0] and the vector [0, 1], the dimension K of the i-th target bounding box feature is the same as the category label c i 0 vector, that is,

[0094] Then, the category label c of the i-th target bounding box feature is spliced with the 0 vector of the same dimension, and the fine-grained domain label i

[0095] For source domain data, the fine-grained domain label of the i-th target bounding box feature For target domain data, the fine-grained domain label of the i-th target bounding box feature Wherein, The dimension of the category label of the target bounding box feature is K, The dimension of the fine-grained domain classification score of the target bounding box feature is 2K.

[0096] Through the above embodiment, the problem that the fine-grained feature is not reasonably aligned in the prior art solution can be solved.

[0097] Step 3, according to the target bounding box features of the source domain and the target domain, the fine-grained domain classification scores of the target bounding box features of the source domain and the target domain are obtained through the fine-grained domain classifier. In the training process of the target detection model, the fine-grained domain classifier constructed is supervised by using the fine-grained domain labels of the target bounding box features, which will help to realize the cross-domain adversarial alignment of the fine-grained features.

[0098] In one embodiment, step 3 specifically includes:

[0099] Step 31, the ordinary domain classifier output layer is respectively expanded to K neurons at each neuron, and the fine-grained domain classifier is constructed.

[0100] As Figure 3 ​As shown, the general domain classifier can only output binary prediction scores, and after being supervised by coarse-grained domain labels, the general domain classifier finally only confuses source domain and target domain features at the overall level. Since the construction of the general domain classifier ignores the class information which is crucial for domain adaptation, the potential fine-grained features in the confused source domain and target domain data may not be properly aligned, which leaves room for the improvement of the performance of the cross-domain target detection model. Considering the effectiveness of adversarial training for cross-domain feature alignment, the present application incorporates class information into the general domain classifier, and then completes the construction process of the fine-grained domain classifier.

[0101] As shown in Figure 4 In order to let the domain classifier align the potential fine-grained features while aligning the overall features of the source domain and target domain data, the present application expands each neuron of the output layer of the general domain classifier to K neurons respectively. The fine-grained domain classifier includes M2 fully connected layers, and except for the M2th fully connected layer (i.e. the output layer), the first (M2-1) fully connected layers are followed by other forms of activation functions such as ReLU, Leaky ReLU, etc. The specific value of M2 needs to be determined according to the actual situation. Let N 2，i represent the number of neurons in the i-th fully connected layer in the fine-grained domain classifier, where i = 1, 2, …, M2.

[0102] Taking M2 = 3 as an example, the number of neurons in the first fully connected layer in the fine-grained domain classifier is N 2，1 = N1 / 2, the number of neurons in the second fully connected layer is N 2，2 = N 2，1 / 2, and the number of neurons in the third fully connected layer is N2,3 = 2K.

[0103] As a person skilled in the art should understand, a gradient reversal layer is set between the fine-grained domain classification and the target detection model, which can be used to establish an adversarial training relationship between the fine-grained domain classifier and the target detection model. Therefore, the fine-grained domain classifier with the above network structure not only focuses on distinguishing whether the input data comes from the source domain or the target domain, but also focuses on distinguishing the specific category of the fine-grained features from the source domain and target domain data. That is to say, when the target detection model combined with the fine-grained domain classifier is trained, the fine-grained domain classifier can model the class probability distribution of the potential fine-grained features in the source domain and target domain data, and in the process of narrowing down the overall distribution of the source domain and target domain data, it can further encourage the adversarial alignment of the fine-grained features of the same category in the source domain and target domain.

[0104] Step 32, taking the features in the i-th source domain or target domain target bounding box as the input of the fine-grained domain classifier, then the fine-grained domain classifier will predict the fine-grained domain classification scores for these target bounding box features.

[0105] In this embodiment, the set composed of the fine-grained domain classification scores of the features within the target bounding boxes of the source domain is denoted as The set composed of the fine-grained domain classification scores of the features within the target bounding boxes of the target domain is denoted as Wherein, denote the fine-grained domain classification score of the i-th feature within the target bounding box of the source domain and the fine-grained domain classification score of the i-th feature within the target bounding box of the target domain, respectively, It is shown that the dimensions of the fine-grained domain classification scores of the features within the target bounding boxes of the source domain and the target domain are both 2K.

[0106] For the source domain data, the fine-grained domain classification score of the i-th feature within the target bounding box is For the target domain data, the fine-grained domain classification score of the i-th feature within the target bounding box is Wherein, denote the j'-th element of the fine-grained domain classification score of the i-th feature within the target bounding box of the source domain, denote the j'-th element of the fine-grained domain classification score of the i-th feature within the target bounding box of the target domain, j' = 1, 2, …, 2K.

[0107] Step 4, calculate the fine-grained feature adversarial alignment loss according to the fine-grained domain label and the fine-grained domain classification score.

[0108] In one embodiment, step 4 specifically includes:

[0109] Step 41, convert the fine-grained domain classification score into a fine-grained domain probability score, and use the fine-grained domain probability score and the fine-grained domain label to calculate the fine-grained feature adversarial alignment loss.

[0110] The j'-th element of the fine-grained domain probability score of the i-th feature within the target bounding box of the source domain is obtained by performing softmax processing on the fine-grained domain classification score of the i-th feature within the target bounding box of the source domain through formula (5), and the j'-th element of the fine-grained domain probability score of the i-th feature within the target bounding box of the target domain is obtained by performing softmax processing on the fine-grained domain classification score of the i-th feature within the target bounding box of the target domain through formula (6).

[0111]

[0112]

[0113] ​​The fine-grained feature adversarial alignment loss adopts the form of multi-class cross-entropy loss function. For each element in the fine-grained domain probability score of the feature in the target bounding box of each source domain, it is multiplied by the corresponding fine-grained domain label after passing through the log function. Similar operations are performed on each element in the fine-grained domain probability score of the feature in the target bounding box of all source domains, and the operation results are sequentially added. After taking the average of the sum, the fine-grained feature adversarial alignment loss of the feature in the target bounding box of the source domain can be obtained As shown in formula (7). The fine-grained feature adversarial alignment loss of the feature in the target bounding box of the target domain is calculated in the same way, as shown in formula (8).

[0114]

[0115]

[0116] In one embodiment, step 5 specifically includes:

[0117] The fine-grained feature adversarial alignment loss is combined into the loss of the target detection model itself. In formula (9), the sum of the fine-grained feature adversarial alignment losses of the features in the target bounding boxes of the source domain and the target domain is adjusted by using the coefficient α, and then added to the target detection model loss above, so as to obtain the total loss of the target detection model combined with the fine-grained domain classifier wherein the specific value of α needs to be determined by model tuning. Generally, 0.01, 0.1, etc. can be taken.

[0118]

[0119] Take formula (9) as the optimization target to train the target detection model combined with the fine-grained domain classifier. When convergence is completed, the optimal model weight in this training process is selected. As a person skilled in the art should understand, the optimal model weight refers to the model weight that makes the target detection model combined with the fine-grained domain classifier have the highest mean average precision (mAP) on the unlabeled target domain. In the inference stage, only the above optimal model weight needs to be loaded for the target detection model (not combined with the fine-grained domain classifier), so that adaptive cross-domain target detection can be realized without additional inference time.

[0120] In the adversarial training process of the fine-grained domain classifier and the target detection model, the fine-grained domain classifier not only focuses on aligning the overall features of the source domain data and the target domain data, but also focuses on aligning the potential fine-grained features in the source domain data and the target domain data. Therefore, the fine-grained features belonging to different categories can also be well aligned across domains. This alleviates the difficulty of the target detection model in classifying different categories of target instances, improves the classification accuracy of the target detection model after locating the target instances, and enables the target detection model to achieve further performance improvement in the cross-domain application process.

[0121] As shown in Figure 5 The fine-grained feature adversarial alignment system for cross-domain target detection provided by the embodiment of the present application includes a source domain and target domain target bounding box feature extraction module, a source domain and target domain fine-grained domain label synthesis module, a source domain and target domain fine-grained domain classification score acquisition module, and a fine-grained feature adversarial alignment loss calculation module, wherein:

[0122] The source domain and target domain target bounding box feature extraction module is used to extract features from the source domain and target domain local feature maps intercepted using the target detection model positioning result, thereby providing the source domain and target domain fine-grained domain classification score acquisition module with source domain and target domain target bounding box features. The module includes 1 pooling layer capable of converting feature maps with different heights and widths into feature maps with the same height and width, and M1 "fully connected layer-activation layer" combinations. The number of neurons in the fully connected layer is N1, and the activation layer can use ReLU, Leaky ReLU, or other forms of activation functions. M1 can be set according to actual conditions, and the default is 2. N1 takes an exponential power of 2, such as 512, 1024, 2048, 4096, etc.

[0123] The source domain and target domain fine-grained domain label synthesis module is used to first convert the classification results of the target detection model into class labels corresponding to the source domain and target domain target bounding box features, and then combine the class labels of the target bounding box features with the coarse-grained domain labels into corresponding fine-grained domain labels. The conversion process from the classification results to the class labels can be realized by one of the following two ways: directly performing softmax processing on the class classification scores of the source domain and target domain target bounding box features, and then converting the obtained class probability scores into class hard labels (hard label); or first adjusting the softmax distribution of the class classification scores of the source domain and target domain target bounding box features using a temperature coefficient, and then using the adjusted class probability scores as class soft labels (soft label). The fine-grained domain label is obtained by concatenating the class label of the target bounding box feature with a 0 vector of the same dimension.

[0124] The source domain and target domain fine-grained domain classification score obtaining module is configured to obtain fine-grained domain classification scores of features in the source domain and target domain target bounding boxes, thereby providing inputs for the fine-grained feature adversarial alignment loss calculation module. The module includes one gradient reversal layer and one fine-grained domain classifier. The fine-grained domain classifier includes M2 fully connected layers, and the total number of neurons in the output layer is 2K. Wherein, K represents the total number of target categories to be detected in the target detection task. Except for the M2th fully connected layer (i.e. the output layer), the first (M2-1) fully connected layers are followed by other forms of activation functions such as ReLU, Leaky ReLU, and the specific value of M2 needs to be determined according to the actual situation. N 2，i represents the number of neurons in the i-th fully connected layer in the fine-grained domain classifier, wherein i = 1, 2, …, M2. Taking M2 = 3 as an example, the number of neurons in the first fully connected layer in the fine-grained domain classifier is N 2，1 = N1 / 2, the number of neurons in the second fully connected layer is N 2，2 = N 2，1 / 2, and the number of neurons in the third fully connected layer is N 2，3 = 2K.

[0125] The fine-grained feature adversarial alignment loss calculation module is configured to calculate the fine-grained feature adversarial alignment loss using the fine-grained domain labels provided by the source domain and target domain fine-grained domain label synthesis module and the fine-grained domain classification scores provided by the source domain and target domain fine-grained domain classification score obtaining module, thereby providing an optimization target for the fine-grained feature adversarial alignment of cross-domain target detection.

[0126] In the adversarial training process of the target detection model, this fine-grained feature adversarial alignment system for cross-domain target detection can focus on aligning the overall features of the source domain and target domain data while also focusing on aligning the potential fine-grained features in the source domain and target domain data. In the inference stage, only this fine-grained feature adversarial alignment system for cross-domain target detection needs to be removed, and after loading the above optimal model weights into the target detection model (without combining the fine-grained domain classifier), adaptive cross-domain target detection can be achieved without additional inference time.

[0127] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application, and are not limiting. Those skilled in the art should understand that the technical solutions described in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A fine-grained feature adversarial alignment method for cross-domain object detection, characterized in that, Comprise: Step 1, according to the positioning result of the target detection model, the source domain and the target domain are extracted from the positioning result in the target bounding box feature; Step 2, according to the classification result of the target detection model, first, the class label corresponding to the source domain and the target domain target bounding box feature is obtained, and then the class label and the coarse-grained domain label are combined to obtain the fine-grained domain label; Step 3, according to the source domain and the target domain target bounding box feature, the fine-grained domain classification score of the source domain and the target domain target bounding box feature is obtained through the fine-grained domain classifier; Step 4, according to the fine-grained domain label and the fine-grained domain classification score, the fine-grained feature is calculated. The alignment loss of the confrontation is calculated; Step 3 specifically comprises: Step 31, respectively expand each neuron of the common domain classifier output layer to K neurons, and construct a fine-grained domain classifier including: M2 fully connected layers, of which the first (M2-1) fully connected layers are followed by an activation function, and the last one is followed by a softmax function. 2，i N represents the number of neurons in the i-th fully connected layer of the fine-grained domain classifier, where i = 1, 2, …, M2; Step 32, taking the features within the i-th source domain or target domain target bounding box as the input of the fine-grained domain classifier, the fine-grained domain classifier predicts that the fine-grained domain classification score of the i-th source domain target bounding box feature is the fine-grained domain classification score of the i-th target domain target bounding box feature wherein, denotes the fine-grained domain classification score of the i-th source domain target bounding box feature the j'-th element of denotes the fine-grained domain classification score of the i-th target domain target bounding box feature the j'-th element of, j' = 1, 2, …, 2K; Step 4 specifically comprises: The softmax processing is performed on the fine-grained domain classification scores of the features in the i-th source domain target bounding box and the fine-grained domain classification scores of the features in the i-th target domain target bounding box by formula (5) and formula (6) respectively to obtain the j'-th element in the fine-grained domain probability score of the features in the i-th source domain target bounding box and the j'-th element in the fine-grained domain probability score of the features in the i-th target domain target bounding box The fine-grained feature adversarial alignment loss of the features in the source domain target bounding box is calculated by using formula (7) The fine-grained feature adversarial alignment loss of the features in the target domain target bounding box is calculated by using formula (8) 2. The fine-grained feature counter-fitting alignment method for cross-domain object detection of claim 1, wherein, The method of "combining the class label and the coarse-grained domain label into the fine-grained domain label" in step 2 specifically comprises: The coarse-grained domain label corresponding to the source domain data is set to [1, 0], and the coarse-grained domain label corresponding to the target domain data is set to [0, 1]; transforming the 0 element in the coarse-grained domain label vector [1, 0] and vector [0, 1] into a dimension K and the class label c of the feature within the i-th target bounding box i same 0 vector; a class label c of the i-th target bounding box inside feature i decomposes into according to its dimension K and spliced with a 0 vector of the same dimension, to obtain a fine-grained domain label of the i-th source domain target bounding box inside feature a fine-grained domain label of the i-th target domain target bounding box inside feature wherein, each of the dimensions of the class label representing the class of the feature within the target bounding box is K, each of the dimensions of the fine-grained domain classification score representing the fine-grained domain classification of the feature within the target bounding box is 2K.

3. The fine-grained feature counter-fitting alignment method for cross-domain object detection of claim 2, wherein, The "class label corresponding to the source domain and the target domain target bounding box feature" in step 2 is realized by one of the following two ways: First, the class classification score of the i-th target bounding box feature is processed by using formula (1) to obtain the class probability score: In the formula, p i，j , y i，j , y i，k represents the jth element of the class probability score p i of the feature in the ith target bounding box, the jth element and the kth element of the class classification score y i , j = 1, 2, …, K; Then, the element p i，j The class label c of the i-th target bounding box inside feature corresponding to the maximum i The j-th element c in the middle i，j is set to 1, and other elements in the class label c i is set to 0, so as to obtain the class hard label of the i-th target bounding box inside feature; Second, the smoothing degree of the softmax distribution of the class classification score of the i-th target bounding box feature is adjusted by using formula (3) to describe the temperature coefficient T, and the class probability score is obtained: As shown in equation (4), the class probability score p of the feature within the target bounding box is calculated i As a class soft label: c i，j = p i，j (4).

4. The method for fine-grained feature counter-fitting alignment for cross-domain object detection of claim 1, wherein, Further comprising: Step 5, fine-grained feature alignment loss of source domain and target domain features in target bounding box The loss is combined into the loss of the target detection model itself, and the target detection model combined with the fine-grained domain classifier is trained to obtain optimal model weight for adaptive cross-domain target detection.

5. The fine-grained feature counter-fitting alignment method for cross-domain target detection of claim 4, wherein, Step 5 specifically comprises: In formula (9), the sum of the fine-grained feature confrontation alignment losses of the features in the source domain and the target domain target bounding box is adjusted by using the coefficient a, and then added to the target detection model loss Therefore, the total loss of the target detection model combined with the fine-grained domain classifier is obtained With formula (9) as the optimization target, the target detection model combined with the fine-grained domain classifier is trained, and when After convergence is completed, the optimal model weight in the current training process is selected.

6. A fine-grained feature adversarial alignment system for cross-domain object detection implementing the method of any one of claims 1-5, characterized in that, Comprise: The source domain and the target domain target bounding box feature extraction module is used for extracting the source domain and the target domain target bounding box feature from the positioning result according to the positioning result of the target detection model; The source domain and the target domain fine-grained domain label synthesis module is used for first obtaining the class label corresponding to the source domain and the target domain target bounding box feature according to the classification result of the target detection model, and then combining the class label and the coarse-grained domain label into the fine-grained domain label; The source domain and the target domain fine-grained domain classification score acquisition module is used for obtaining the fine-grained domain classification score of the source domain and the target domain target bounding box feature through the fine-grained domain classifier according to the source domain and the target domain target bounding box feature; The fine-grained feature is calculated. The alignment loss of the confrontation is calculated.

7. The fine-grained feature counter-factual alignment system for cross-domain target detection of claim 6, wherein, The fine-grained feature adversarial alignment loss calculation module is further configured to adjust the sum of the fine-grained feature adversarial alignment losses of the features in the source domain and the target domain target bounding boxes by using the coefficient α in formula (9), and then add the adjusted sum to the known target detection model loss Thus, the total loss of the target detection model combined with the fine-grained domain classifier is obtained The target detection model combined with the fine-grained domain classifier is trained with formula (9) as the optimization target, and when After the convergence is completed, the optimal model weight in the current training process is selected. In the formula: is the fine-grained feature alignment loss of the features in the source domain target bounding box calculated by formula (7), is the fine-grained feature alignment loss of the features in the target domain target bounding box calculated by formula (8): wherein, is the j'th element of the fine-grained domain probability score of the i'th source domain target bounding box inside feature, is the j'th element of the fine-grained domain probability score of the i'th target domain target bounding box inside feature, is the j'th element of the fine-grained domain label of the i'th source domain target bounding box inside feature, is the j'th element of the fine-grained domain label of the i'th target domain target bounding box inside feature, s , n t are the total number of source domain and target domain local feature maps respectively, K is the dimension of the class label of the source domain and target domain target bounding box inside feature, j' = 1, 2, …, 2K.

Images