Foreground-based and semantic-independent cross-domain object detection method and system

Abstract

This application discloses a cross-domain object detection method based on foreground activation and semantic independence. First, a multi-scale feature aggregation and alignment module is used to capture and aggregate image-level features of different scales in the feature extraction layer for adversarial training. Then, a foreground attention map of the training image is output by a foreground attention activation module and input into a foreground region weighted alignment module to weight and align the image-level aggregated features. Finally, a single instance feature of the target object is extracted and input into a category-semantic-independent domain discriminator. This guides the instance features to achieve semantic separation and independence of different category instance features at each prediction pipeline of the discriminator. Adversarial training of the feature domain is performed at the corresponding category prediction output pipeline to enhance the inter-domain confusion of similar instance features. This enables domain adaptive training of foreground region weighting and category-semantic independence for a single-stage object detector, achieving robust knowledge transfer of the detector at the foreground region and category semantic levels.

Description

Technical Field

[0001] This application belongs to the field of cross-domain object detection technology based on deep learning, specifically involving a cross-domain object detection method and system based on foreground activation and semantic independence. Background Technology

[0002] Deep learning-based visual object detection technology far surpasses traditional hand-crafted feature-based detection methods in terms of detection accuracy, and has been widely used in video surveillance, intelligent security, smart detection, and autonomous driving. However, these algorithms only demonstrate extremely high detection accuracy on publicly available datasets. When deployed to perform detection tasks in real-world scenarios, their detection performance degrades significantly. This is because the training and validation data for deep learning-based object detection algorithms come from the same dataset, essentially conforming to the assumption of independent and identically distributed data. However, real-world scene data is often more complex and diverse, resulting in a data distribution shift between the data used for training the object detector and the actual detection data. This domain shift phenomenon leads to a severe degradation in the detection performance of the object detector, limiting the application of excellent object detection algorithms in real-world scenarios. While it is possible to improve detection performance by training networks with a large amount of image data from real-world application scenarios, the instance-level annotation of the training data is extremely complex and costly in engineering applications, making it impractical.

[0003] Domain-adaptive object detection models built using transfer learning-based unsupervised domain adaptation theory can be trained on both source domain data with instance-level annotations and target domain data available only as image data. This transfers object detection knowledge from the source domain to the target domain, resulting in excellent detection performance in the target domain. In recent years, domain adaptation techniques have been widely applied to various computer vision tasks such as image classification, object detection, semantic segmentation, and instance segmentation, achieving good results. Because object detection is more complex than image classification, developing domain-adaptive visual object detection techniques is more challenging. Summary of the Invention

[0004] This invention discloses a cross-domain target detection method and system based on foreground activation and semantic independence, which can effectively solve at least one of the technical problems involved in the background art.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows:

[0006] A cross-domain object detection method based on foreground activation and semantic independence includes the following steps:

[0007] Step 1: Obtain the source domain dataset and the target domain dataset, and build a basic single-stage object detection model;

[0008] Step 2: On the basic single-stage object detection model, set up a multi-scale feature aggregation and alignment module, a foreground attention activation module, a foreground region weighted alignment module, an instance feature acquisition module, and an instance feature semantically independent alignment module, which together form a single-stage cross-domain object detection model based on foreground activation and semantic independence.

[0009] Step 3: Obtain the total loss of the single-stage cross-domain object detection model based on foreground activation and semantic independence, and train it iteratively for multiple rounds. Use the trained single-stage cross-domain object detection model based on foreground activation and semantic independence in the target domain for cross-domain object detection.

[0010] As a preferred improvement of the present invention, step 1, obtaining the source domain dataset and the target domain dataset, specifically includes:

[0011] Select labeled datasets that are similar to actual application scenarios and have sufficient data volume as source domain datasets, and collect unlabeled data from actual application scenarios to form target domain datasets;

[0012] The construction of the basic single-stage target detection model specifically includes:

[0013] The YOLOv3 single-stage object detection model is used as the basic object detector, and Darknet-53 is used as the feature extraction network to extract image features. The image features are fed to three object detection heads (large, medium, and small) through a three-layer branch structure. After processing by the detection heads, the object detection results are obtained. YOLOv3 calculates the detection loss of the basic single-stage object detection model based on the object detection results.

[0014] As a preferred improvement of the present invention, step 2, which involves setting a multi-scale feature aggregation and alignment module on the basic single-stage target detection model, specifically includes:

[0015] Extract output features from different network layers;

[0016] Aggregate the output features to obtain aggregated features;

[0017] Construct an aggregated feature domain discrimination network to predict the domain category probability value of aggregated features;

[0018] Calculate the aggregated feature alignment loss

[0019] As a preferred improvement of the present invention, in step 2, a foreground attention activation module and a foreground region weighted alignment module are set on the basic single-stage object detection model, specifically including:

[0020] Obtain the foreground attention activation map at the detection head;

[0021] The foreground attention activation map is normalized, filtered, and fused to obtain the foreground attention overlay map.

[0022] Weighted aggregated features are obtained by using foreground attention overlay map and aggregated features, and then input into the weighted feature domain discriminant network for domain adversarial training.

[0023] Calculate the weighted alignment loss for weighted aggregated feature adversarial training.

[0024] As a preferred improvement of the present invention, an instance feature acquisition module and an instance feature semantically independent alignment module are set on the basic single-stage object detection model, specifically including:

[0025] Obtain instance characteristics of the source and target domains;

[0026] Construct a category-semantically independent domain discriminant to predict the category and domain to which instance features belong;

[0027] Based on the predicted vectors of instance features, calculate the semantically independent loss for different categories. Alignment loss between domains of the same category

[0028] As a preferred improvement of the present invention, the acquisition of instance features of the source domain and the target domain specifically includes:

[0029] Collect the bounding box parameters of the source domain annotation, separate the bounding boxes and confidence scores from the detection results obtained from the three detectors corresponding to the target domain training data, and use non-maximum suppression to filter out the bounding box parameters with high confidence scores from the three detectors. Perform ROIAlign operation on the bounding box parameters of the source and target domains and the corresponding aggregated features to obtain the instance features of the source and target domains.

[0030] As a preferred improvement of the present invention, in step 3, the total loss of the single-stage cross-domain object detection model based on foreground activation and semantic independence is... The loss consists of the detection loss of the basic single-stage object detection model, the loss of the multi-scale feature aggregation and alignment module, the loss of the foreground region weighted alignment module, and the loss of the instance feature semantically independent alignment module, as shown in the following expression:

[0031]

[0032] In the formula, This represents the detection loss of the basic single-stage object detection model. This represents the aggregated feature alignment loss. Indicates the weighted alignment loss. This represents semantically independent loss for different categories. λ1, λ2, λ3, and λ4 represent the alignment loss between domains of the same category, and are the balance coefficients of each loss, ranging from 0.1 to 1.

[0033] As a preferred improvement of the present invention, in step 3, the single-stage cross-domain target detection model based on foreground activation and semantic independence is trained in multiple rounds using the total loss. After training, the domain adaptive model weights are saved and loaded into the basic single-stage target detection model to complete cross-domain target detection in the target domain.

[0034] This invention also provides a cross-domain object detection system based on foreground activation and semantic independence, comprising a data acquisition module, a training module, and an inference module, wherein...

[0035] The data acquisition module is used to collect data from actual application scenarios and publicly available large datasets related to the detection task. After preprocessing the data, it packages the data into source domain data and target domain data that can be used by the training module. At the same time, it provides the inference module with task data that requires real-time target detection.

[0036] The training module is used to transfer object detection knowledge from the source domain to the target domain using foreground attention-guided and category semantic independence methods.

[0037] The inference module is used to receive real-world scene data, perform real-time inference on image data, and output target detection results.

[0038] As a preferred improvement of the present invention, the network of the training module is a single-stage cross-domain object detection model based on foreground activation and semantic independence, and the network of the inference module is a basic single-stage object detection model.

[0039] The beneficial effects of this invention are as follows:

[0040] 1. The multi-scale feature aggregation and alignment module provided in this application aggregates the output features of different sizes at multiple levels of the backbone network and inputs them into the domain discriminant network to complete adversarial training on the aggregated features of the source domain and the target domain. This can simultaneously avoid the problems of insufficient feature alignment and redundant model parameters, and guide the target detection model to perform preliminary domain adaptation at the image feature level.

[0041] 2. The foreground attention activation module and foreground region weighted alignment module provided in this application weight the aggregated features by obtaining the foreground region activation map from the high-level features of the detection head, and inputting it into the weighted feature domain discriminant network for domain adversarial training. This can enhance the foreground region features in the aggregated features, avoid suboptimal results caused by overtraining the background region, guide the target detection network to focus on the domain adaptation of important foreground regions, and further improve the transfer robustness.

[0042] 3. The instance feature acquisition module and instance feature semantically independent alignment module provided in this application can obtain a single target instance feature vector from aggregated features using source domain labeled bounding boxes and target domain predicted bounding boxes. The vector is then input into a category semantically independent domain discrimination network. Based on category semantically independent loss and adversarial training loss of the same type of features, the semantic separation and independence of different category features and the enhancement of confusion between the same type of feature domains are achieved simultaneously. This avoids the problem of impaired detection performance caused by misalignment of different semantic instance features and guides the semantic independence of cross-domain object detection models at the category level. Attached Figure Description

[0043] Figure 1 A network structure diagram provided for embodiments of this application;

[0044] Figure 2 A flowchart illustrating the method provided in this application embodiment;

[0045] Figure 3 This is a system framework diagram provided for an embodiment of this application. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0047] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.

[0048] Furthermore, in this invention, descriptions involving "first," "second," etc., are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0049] In this invention, unless otherwise explicitly specified and limited, the terms "connection," "fixed," etc., should be interpreted broadly. For example, "fixed" can mean a fixed connection, a detachable connection, or an integral part; it can mean a mechanical connection or an electrical connection; it can mean a direct connection or an indirect connection through an intermediate medium; it can mean the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0050] Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but only if they are based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0051] Please see Figures 1 to 3 As shown, this application proposes a cross-domain object detection method based on foreground activation and semantic independence, including the following steps:

[0052] Step 1: Obtain the source domain dataset and the target domain dataset, and build a basic single-stage object detection model.

[0053] Specifically, a labeled dataset with sufficient data volume and similarity to the actual application scenario is selected as the source domain dataset, and unlabeled data from the actual application scenario is collected to form the target domain dataset. The subscripts s and t are used to represent the source domain and target domain, respectively, with the source domain being... Target domain is in, This represents the image data of the i-th sample in the source domain. Let c represent the bounding box information of the i-th sample in the source domain and its category c, where c∈{1,2,…,K} and K represents the total number of categories. N represents the image data of the j-th sample in the target domain. s N t These represent the total number of samples in the source domain and the total number of samples in the target domain, respectively. It is important to note that the source and target domains should maintain the same target detection category.

[0054] Since the cross-domain object detection model designed based on the two-stage object detection model Faster RCNN has a complex network structure and slow detection speed, this invention selects a single-stage object detection model based on deep learning as the basic object detector of the entire cross-domain object detection network. This facilitates the subsequent embedding of various domain adaptive modules in this framework. The network structure of the basic object detector provides the cross-domain detection model with preliminary feature extraction and object detection functions.

[0055] To facilitate the subsequent explanation of the model building process, this invention selects the classic single-stage object detector YOLOv3 as the basic object detector. The YOLOv3 object detection network uses Darknet-53 as the feature extraction network to extract image features from the input image. These image features are then fed to large, medium, and small object detection heads through a three-layer branch structure. The detection heads process these features into object detection results with unchanged height and width and a channel count of (number of categories c+5) × number of anchor boxes. The channel values ​​at each location on the feature map represent the x, y, z, and h coordinate offsets of the three anchor templates, the confidence parameter, and the score for each category. YOLOv3 calculates the model's detection loss based on this prediction information. The detection loss of the basic object detector is denoted as... The expression is shown in equation (1) below:

[0056]

[0057] In the formula, For target bounding box offset loss, Classification loss for the target, The target confidence loss.

[0058] Those skilled in the art should understand that when building a single-stage cross-domain object detection model based on foreground activation and semantic independence, the basic single-stage object detector selected only needs to have similar functional modules and network structure to YOLOv3, and should not be limited to YOLOv3.

[0059] Step 2 involves setting up a multi-scale feature aggregation and alignment module, a foreground attention activation module, a foreground region weighted alignment module, an instance feature acquisition module, and an instance feature semantically independent alignment module on the basic single-stage object detection model. These modules together form a single-stage cross-domain object detection model based on foreground activation and semantic independence. Specifically, this includes the following steps:

[0060] Step 21: Set up a multi-scale feature aggregation and alignment module on the basic single-stage target detection model;

[0061] The basic object detector built in step 1 already possesses object detection capabilities, but it is a supervised object detector and cannot be trained using unlabeled object domain data, resulting in extremely poor detection performance in the object domain. Generally, image-level domain adaptation modules in domain adaptation methods either design a domain discriminant model for only the output features of a certain layer of the feature extraction layer to complete domain adversarial training, which carries the risk of insufficient feature alignment; or they design multiple domain discriminators with similar structures at different layers of the network, leading to redundant parameters and low training efficiency in the domain adaptation object detection model.

[0062] To avoid insufficient feature alignment and redundant model parameters, this invention constructs a multi-scale feature aggregation and alignment module, which simultaneously possesses the functions of multi-scale feature aggregation and alignment. In one round of iterative training, output features of different sizes from multiple layers of the backbone network are first aggregated. Then, a domain discriminant network is used to align the multi-scale aggregated features to guide the cross-domain object detection model to perform preliminary domain adaptation at the image feature level. Step 21 specifically includes:

[0063] (a) Extracting output features from different network layers.

[0064] The output feature maps of different network layers have inconsistent sizes. As image features are transmitted from front to back in the network, the size of the feature maps decreases, weakening fine-grained information and potentially causing the loss of feature information for some small targets. Therefore, based on the characteristics of the YOLOv3 detection network, the output features of the three residual blocks after the Darknet-53 feature extraction layer are extracted as the input of the module, denoted as the source domain image-level features. and target domain image-level features in, H i W i C i These represent the height, width, and number of channels of the feature, respectively. It's important to note that this only applies to the YOLOv3 object detector; for other basic object detector networks, any reasonable output feature map can be used as input to this module.

[0065] (ii) Aggregate the output features to obtain aggregated features.

[0066] In one iteration of training, the input module has a total of 6 features, including 3 source domain features and 3 target domain features. These features come from different network layers, so their height and width vary, exhibiting a size ratio relationship of H1 = 2H2 = 4H3. Therefore, the multi-scale feature aggregation and alignment module first adjusts the feature sizes to be consistent and then stacks them along the channel dimension to complete feature aggregation. Perform 2x and 4x upsampling operations. and The height, width, and number of channels become 2H2, 2W2, C2 and 4H3, 4W3, C4 respectively. The height and width have been adjusted to be consistent, and finally, they are stacked sequentially along the channel dimension to form the source domain aggregate feature. Aggregate features with target domain in The height, width, and number of channels are H1, W1, and (C1+C2+C3), respectively. Since the upsampling operation does not destroy the feature structure information, the aggregated features have complex semantic information and multi-scale target information.

[0067] (III) Constructing an aggregated feature domain discrimination network.

[0068] The function of the aggregated feature discriminator network is to output a predicted value for the domain category of the aggregated features using an adversarial training strategy. Theoretically, features belonging to the source domain should be predicted as 1, and features belonging to the target domain should be predicted as 0. Before being input into the discriminator network, the aggregated features pass through a gradient inversion layer, which inverts the sign of the gradient flowing through them, thus forming an adversarial training network architecture between the backbone network and the domain discriminator network.

[0069] The aggregated feature domain discriminant network consists of three 3×3 convolutional layers and one fully connected layer. In the domain discriminant network, the aggregated features first pass through a gradient inversion layer, and then through three 3×3 convolutional layers to compress the number of channels from (C1+C2+C3) to 64. Then, the output is flattened into a one-dimensional feature vector and input into the fully connected layer. Finally, a domain class probability value is predicted for each aggregated feature, which is used to calculate the domain alignment loss, as shown in Equations (2) and (3).

[0070]

[0071]

[0072] In the formula, This represents the predicted value of the aggregated features in the source domain. This represents the aggregated feature prediction value of the target domain, where i and j represent the indices of the corresponding images in the dataset, i∈{1,2,3,…,N}. s}, j∈{1,2,3,…,N t}, E1(·) represents the aggregated feature domain discriminant network.

[0073] (iv) Calculate the aggregation special alignment loss

[0074] Since some samples in the source and target domains are far apart in the feature distribution space and are distributed on both sides of the domain boundary, these samples are easy for the domain discrimination network to discriminate and account for a large proportion of the dataset. Even if the discrimination of these "easy samples" is good, it is difficult to further improve the domain alignment effect. Therefore, the focal loss (FL) is used to calculate the domain alignment loss of aggregated features, so that the network focuses on processing the difficult-to-align samples and reduces the weight of the "easy sample" loss. The specific calculation process is shown in Equations (4), (5) and (6):

[0075]

[0076]

[0077]

[0078] In the formula, This represents the source domain aggregation feature alignment loss. This represents the target domain aggregated feature alignment loss. The aggregated feature alignment loss is backpropagated and used to activate features invariant in the output domain of the feature extraction network via a gradient inversion layer, thus initially completing the image-level domain adaptation of the target detection model.

[0079] Step 22: Set the foreground attention activation module and the foreground region weighted alignment module on the basic single-stage object detection model;

[0080] In images used for object detection, the region containing the target is the foreground region, and the region surrounding the target is the background region. Although the background region can provide some environmental semantic information for object detection, the foreground region is clearly more important because it directly contains core information such as the shape, texture, and color of the detected target. However, general domain-adaptive object detection methods only use a domain classifier to uniformly align image-level features, without distinguishing the alignment strength between the foreground and background regions of the image. This leads to suboptimal image-level domain adaptation results.

[0081] Therefore, this invention sets up a foreground attention activation module to obtain an interest foreground activation map of the target region from the high-level features of the detection heads. The high-level features of the three detection heads (large, medium, and small) of the YOLOv3 detector already encode the location region information of the detected target. The foreground attention activation module outputs three target foreground maps at the large, medium, and small scales corresponding to these three detection heads, and these maps are fused together to obtain a fused foreground map. The fused foreground map is then aligned with the aggregated features obtained in step 21 at the image level using a foreground region-guided secondary weighted feature alignment to guide the network to apply more attention to the domain adaptively of the target foreground region. Step 22 specifically includes:

[0082] (a) Obtain the foreground attention activation map at the detection head.

[0083] In one iteration of training, high-level features at three detector heads are first obtained, with the source domain detector head features denoted as... The target domain detection head features are denoted as The superscripts s and t indicate that the feature belongs to the source domain or the target domain, and the subscripts d, m, and s represent large, medium, and small detection heads, respectively. H, W, and C represent the height, width, and number of channels of the feature, respectively. These six features are processed by the foreground attention activation module to obtain six regions of interest activation maps, using the largest detection head source domain features of the i-th image data. For example, the detection head features are first pooled into activation vectors through a global average pooling layer. As shown in equation (7), where, The dimension of the activation vector is C, which represents the number of channels in the detection head feature:

[0084]

[0085] In the formula, x, y, and k represent the indices of the height, width, and number of channels of the detection head feature, respectively, where x∈{1,2,…,H}. d}, y∈{1,2,…,W d}, k∈{1,2,…,C d}

[0086] Activation vector Each value in the dimension represents a feature of the detection head. The strength of the spatial location information of the foreground target in the corresponding channel. Therefore, the activation vector is... With corresponding detection head features Linear fusion along the channel dimension yields a size of H. d ×W d Foreground attention activation map As shown in equation (8):

[0087]

[0088] In the formula, attention activation map It is a two-dimensional tensor, whose height and width correspond to the height and width of the corresponding detection head features. The activation value at each coordinate on the tensor represents the intensity of the foreground target information at that location. Similarly, equations (7) and (8) can be used to calculate the corresponding foreground attention activation maps for the other five detection head features.

[0089] (ii) The obtained foreground attention activation map of the detection head is normalized, filtered and fused to obtain the foreground attention overlay map.

[0090] The above step (I) only provides a preliminary foreground attention activation map. However, the activation map has not undergone normalization or filtering, and it still contains a lot of noise. Direct use of this map risks confusing the foreground positions. The foreground attention activation map corresponding to the large detector head in the i-th image data is shown below. For example, we first normalize each position on the foreground activation map, as shown in equation (9):

[0091]

[0092] In the formula, (x,y) represents the coordinate position on the foreground activation map, max(·) represents the maximum value operation, and min(·) represents the minimum value operation. At this point, the activation map... The value at each position is restricted to between 0 and 1.

[0093] Then, the normalized foreground attention activation map was analyzed. Calculate the filter factor As shown in equation (10), the filter factor serves as a threshold to determine whether each location in the attention activation map belongs to the foreground or background. For values ​​smaller than the filter factor... The position is directly set to 0, and the value is greater than or equal to the filter factor. The positional activation values ​​remain unchanged, resulting in the processed foreground attention activation map. As shown in equation (11):

[0094]

[0095]

[0096] In the formula, I(·) is the indicator function, which is 1 when the condition is met and 0 otherwise, and α represents the adjustment factor, which is generally taken as 0.4. The remaining foreground attention activation maps are processed using equations (9), (10), and (11) to obtain... and

[0097] Of the six attention activation maps, three belong to the source domain and three to the target domain, containing target foreground information at three different scales from largest to smallest. Using these three scales of foreground attention activation maps separately for feature weighting and alignment would be cumbersome. Therefore, the three scales of foreground attention activation maps can be merged into a single foreground activation map before feature weighting and alignment. Since the height and width dimensions of these three foreground attention activation maps (large, medium, and small) are different, corresponding to the height and width of the output features of the detection head, and considering that the weighted features are the aggregated features from step 21... Therefore, we first use linear interpolation to adjust these three foreground attention activation maps to aggregate features. The height and width are H1×W1. Then, the elements at each position of the three activation maps are summed to obtain the overlay map. Finally, the overlay map is normalized again, as shown in equations (12) and (13):

[0098]

[0099]

[0100] In the formula, This represents the foreground attention overlay map of the i-th source domain image. S[·] represents the foreground attention overlay map of the i-th target domain image, and S[·] represents the sigmoid function normalization. This indicates the operation of linearly interpolating and upsampling the activation graph to H1×W1. This indicates an operation that adds elements one by one at the corresponding positions.

[0101] (iii) Use foreground attention overlay map and aggregate features to obtain weighted aggregate features, and input them into the weighted feature domain discriminant network for domain adversarial training.

[0102] Aggregation features and It is an aggregation of image-level features from the feature extraction layer, containing high-level semantic information and fine-grained information about multi-scale targets. The weights at different positions in the foreground attention overlay map reflect the spatial location information of the foreground region. Therefore, the aggregated features and the foreground attention overlay map are multiplied along the channel dimension to enhance the foreground region features in the aggregated features and suppress the background region features, as shown in Equations (14) and (15):

[0103]

[0104]

[0105] In the formula, The weighted aggregated features of the i-th source domain image, This represents the weighted aggregated features of the i-th target domain image, where parameter β is the background region suppression intensity factor, typically ranging from 0.3 to 0.7. This indicates element-wise multiplication.

[0106] The weighted aggregated features are then input into the weighted feature domain discriminant network E2. After passing through three 1×1 convolutional layers, the number of channels is compressed from (C1+C2+C3) to 1. Finally, a sigmoid function is applied to obtain a domain prediction score map of size H1×W1. The score at each location on the map represents the probability that the image patch in the receptive field at that location belongs to the source domain. This domain classification method using score maps can independently predict the domain category at different locations in the image, which is beneficial for domain adversarial training in the foreground region.

[0107] (iv) Calculate the weighted alignment loss for weighted aggregation feature adversarial training.

[0108] Considering that the weighted feature domain discriminant network E2 outputs a domain prediction score map of size H1×W1, the least squares loss is used to calculate the loss function for weighted aggregated feature adversarial training. Since a loss value is calculated for each position of the domain prediction score map, the loss at each position is weighted again using the foreground attention overlay map, as shown in equations (16), (17), and (18):

[0109]

[0110]

[0111]

[0112] In the formula, Let these represent the source domain weighted alignment loss and the target domain weighted alignment loss, respectively. This represents the output prediction value of the regional discriminator E2 at location (w,h). This represents the weight value of the foreground attention overlay at position (w,h). The foreground attention-weighted aggregated feature alignment loss is backpropagated to guide the object detection network to focus on the domain adaptation of important foreground regions, further improving transfer robustness.

[0113] Step 23: Set up an instance feature acquisition module and an instance feature semantic independent alignment module on the basic single-stage object detection model.

[0114] Instance features in the source and target domains contain feature information of individual targets, allowing for more refined adversarial training between them and further mitigating data distribution offset between the source and target domains. However, previous methods have some drawbacks. First, previous object detection frameworks all had independent Region Generation Networks (RPNs) and ROI layers, and the bounding box detection and target class discrimination of single-stage object detectors were fused together, making it impossible to directly obtain instance features from the target domain. Second, previous instance-level feature alignment simply input all features into the domain discriminator to predict the domain class. This operation did not consider the semantic mismatch caused by the differences in instance features between different classes. The simple domain discriminator network struggled to complete adversarial training between different classes, and forced alignment might degrade the detector's performance.

[0115] Considering that the source domain has available instance-level annotations, true-labeled bounding boxes are used to extract source domain instance features from the aggregated features of the source domain. The target domain lacks annotation information; therefore, high-confidence bounding boxes are separated and selected from the detection results of the target detector head to extract target domain instance features from the aggregated features of the target domain. To address the issue of differences in instance features between different categories, this invention constructs a category-semantic-independent domain discrimination network, using source domain category annotation information and target domain category information predicted by detection to guide training, predicting both the category and the domain for each instance feature. Step 23 specifically includes:

[0116] (a) Obtain the instance characteristics of the source and target domains.

[0117] Specifically, firstly, the bounding box parameters labeled in the source domain are collected. Then, the bounding boxes and confidence scores are separated from the detection results (including bounding box parameters, confidence parameters, and classification parameters) obtained from the three detection heads corresponding to the target domain training data. Non-Maximum Suppression (NMS) is used to filter out the bounding box parameters with high confidence from the three detection heads. The bounding box parameters of the source and target domains are then compared with the corresponding aggregated features. Perform the ROIAlign operation to obtain instance-level features of the source and target domains. In one iteration of training, let the m-th instance feature of the i-th source domain image be denoted as... The feature of the m-th instance of the j-th target domain image is This indicates that the instance features have been flattened into a one-dimensional vector of dimension d.

[0118] (ii) Construct a domain discriminator with semantically independent categories to obtain the prediction vector of instance features.

[0119] Unlike the instance-level domain discriminators in previous domain adaptation methods, the class-semantic-independent domain discriminator must simultaneously predict the class and domain of the input features. Therefore, the class-semantic-independent discriminator E3 uses a fully connected neural network consisting of three hidden layers. Except for the output layer, each layer uses the ReLU activation function and Dropout operation, ultimately outputting 2C predicted probability scores for each instance feature, i.e., a probability prediction vector. Where C represents the total number of categories. The prediction vector is based on the features of the m-th instance of the i-th image in the source domain. For example, the 2C predicted values ​​are grouped into pairs, representing the probability of belonging to the corresponding category as a feature of the source domain and the probability of belonging to the corresponding category as a feature of the target domain, respectively. Since the target domain has no category labeling information, the category predicted by the detection head is used as the pseudo-label of the corresponding instance feature, which, together with the real label of the source domain, guides the classification training of the category semantically independent domain discriminator; the real domain label guides the domain adversarial training of instance features between the two domains.

[0120] (III) Calculating semantic independence loss for different categories Alignment loss between domains of the same category

[0121] The semantic information structures of instance features of different categories differ greatly. Using a single prediction output pipeline will confuse their feature structures, leading to incorrect domain category predictions. Therefore, source domain annotation and target pseudo-labels are used to guide the network in performing category semantic separation. Specifically, the probability of a feature belonging to the c-th category is first separated and normalized from the prediction vector of the corresponding instance feature, as shown in Equation (19):

[0122]

[0123] In the formula, Let δ represent the probability that the m-th feature vector of a training sample belongs to the c-th class. m [2c]、δ m [2c+1] represent the probability that the corresponding feature vector belongs to the source domain and the predicted value that it belongs to the target domain, respectively. This represents the normalization operation. This operation is used to calculate and predict the category to which instance features belong, utilizing multiple output pipelines.

[0124] Then, the semantic independence loss of the categories is calculated using the true labeled categories in the source domain and the pseudo-labels predicted in the target domain. As shown in equation (20):

[0125]

[0126] In the formula, M represents the total number of feature vectors in the input category semantically independent discriminator. This represents the true labeled category or pseudo-label of the corresponding instance feature. Category semantic independence loss. The discriminator parameters of the category semantic independence domain are optimized in reverse, thereby enhancing the discriminator network's class distinguishability of instance feature vectors and achieving category semantic independence between instance features.

[0127] Next, for the alignment of similar features across different domains, the probability that the corresponding feature vector belongs to the source domain is first calculated from the prediction vector, as shown in Equation (21):

[0128]

[0129] In the formula, Let represent the probability that m feature vectors belong to the i-th category and are from the source domain. After obtaining the domain category prediction value, the alignment loss between domains of the same category is calculated according to equation (22).

[0130]

[0131] In the formula, M represents the total number of instance features, and C represents the total number of classes. Loss Function The gradient inversion layer enhances the network's domain confusion of features from instances of the same category.

[0132] Step 3: Obtain the total loss of the single-stage cross-domain object detection model based on foreground activation and semantic independence, and train it iteratively for multiple rounds. Use the trained single-stage cross-domain object detection model based on foreground activation and semantic independence in the target domain for cross-domain object detection.

[0133] Specifically, in the entire training process of the single-stage cross-domain object detection model based on foreground activation and semantic independence, there are two data streams input to the model: a source domain data stream and a target domain data stream. The basic single-stage object detection network is responsible for calculating the output object detection loss on the source domain data. The multi-scale feature aggregation and alignment module, the foreground attention activation module, the foreground region weighted alignment module, the instance feature acquisition module, and the instance feature semantically independent alignment module are jointly responsible for calculating the domain alignment loss using the output features and domain labels of the source and target domains. During the backpropagation of the loss, the object detection loss optimizes the network parameters and improves the network's detection capability, while the domain alignment loss output by the domain adaptation module is used through a gradient reversal layer to incentivize the detection network to output domain-invariant features in an adversarial training manner.

[0134] Therefore, the total loss of the single-stage cross-domain object detection model based on foreground activation and semantic independence is... The function consists of the detection loss of the basic single-stage object detection model, the loss of the multi-scale feature aggregation and alignment module, the loss of the foreground region weighted alignment module, and the loss of the instance feature semantically independent alignment module, as shown in equation (23):

[0135]

[0136] In the formula, This represents the detection loss of the basic single-stage object detection model. This represents the aggregated feature alignment loss. Indicates the weighted alignment loss. This represents semantically independent loss for different categories. λ1, λ2, λ3, and λ4 represent the alignment loss between domains of the same category, and are the balance coefficients of each loss, typically ranging from 0.1 to 1.

[0137] The total loss in equation (23) As the optimization objective of the single-stage cross-domain object detection model based on foreground activation and semantic independence, the model parameters are optimized using the stochastic gradient descent (SGD) optimizer. The single-stage cross-domain object detection model based on foreground activation and semantic independence is trained in multiple rounds of iteration, and the domain adaptive model weights are saved after training.

[0138] All modules designed in this invention are plug-and-play. In inference mode, only the basic single-stage object detector model can be retained without affecting inference speed. Since the saved domain adaptive model weights have undergone domain adversarial training after training, achieving attention guidance for the foreground region and semantically independent alignment of each category, they can be directly loaded into the basic single-stage object detection model, thus exhibiting excellent cross-domain detection performance in the unlabeled object domain.

[0139] This invention also provides a cross-domain object detection system based on foreground activation and semantic independence, comprising a data acquisition module 1, a training module 2, and an inference module 3, wherein...

[0140] The data acquisition module 1 is used to collect data from actual application scenarios and public large datasets related to the detection task. The data is packaged into source domain data and target domain data that can be used by the training module 2 after data preprocessing steps such as cropping, rotation, and color brightness enhancement. At the same time, it provides task data that needs to be detected in real time to the inference module 3.

[0141] The training module 2 is used to transfer object detection knowledge from the source domain to the target domain using foreground attention guidance and category semantic independence. This module can obtain a detection network with robust performance in the target domain using only source domain annotation information. Considering the training memory overhead of this module, it is considered to set it up on a remote server.

[0142] The inference module 3 receives real-world scene data, performs real-time inference on the image data, and outputs target detection results. The inference network structure of the inference module 3 is lightweight and is typically deployed on a local device to perform the detection task; the network parameters are downloaded from the cloud training module.

[0143] It should be further explained that the inference network of the inference module 3 is a basic single-stage object detection model; the training module 2 is a single-stage cross-domain object detection network based on foreground activation and semantic independence provided by this invention. This network is composed of a basic single-stage object detector, a multi-scale feature aggregation and alignment module, a foreground attention activation module, a foreground region weighted alignment module, an instance feature acquisition module, and an instance feature semantically independent alignment module, wherein:

[0144] The basic single-stage object detection model is a pre-selected single-stage object detector that provides basic feature extraction and object detection functions, making it convenient to build other domain adaptive modules on this basis.

[0145] The function of the multi-scale feature aggregation and alignment module is to aggregate the output features of different sizes at multiple levels of the backbone network to obtain aggregated features with complex semantic information and multi-scale target information. Then, the domain discriminant network is used to align the multi-scale aggregated features of the source domain and the target domain, thereby guiding the target detection model to perform preliminary domain adaptation at the image feature level.

[0146] The function of the foreground attention activation module is to activate the high-level features of the detector head to obtain a foreground map of the target region of interest. The high-level features at the detector head encode the location region information of the detected target. The three high-level features of the detector head are input into the foreground attention activation module. After filtering, normalization and other steps, three target foreground region maps of large, medium and small scales are output and fused together to obtain a fused foreground map.

[0147] The function of the foreground region weighted alignment module is to use the fused foreground map to weight the aggregated features, enhance the foreground region features in the aggregated features, suppress the background region features, and construct a weighted feature domain discriminant network for domain adversarial training. This guides the object detection network to focus on the domain adaptation of important foreground regions, further improving transfer robustness.

[0148] The instance feature acquisition module uses the source domain ground truth annotation and the target domain to separate and filter the bounding box parameters from the detection head's detection results and perform ROIAlign operation with the corresponding aggregated features to obtain instance-level feature vectors of the source and target domains.

[0149] The core of the instance feature semantically independent alignment module is a category semantically independent domain discriminant network. It simultaneously predicts the category and domain of the input instance features, calculates the category semantic independence loss and the adversarial training loss of similar features by combining the source domain ground truth annotations and the target domain pseudo labels, guides the instance features to be semantically independent at each prediction pipeline of the discriminator, and performs feature adversarial training at the corresponding prediction output pipeline based on the predicted category of the instance features, thereby enhancing the inter-domain confusion of similar instance features.

[0150] During the entire system operation, the local data acquisition module first uses local devices to collect image data of the actual scene and uploads it to the cloud. The cloud data acquisition module performs a series of preprocessing on these data and public big data datasets to generate source domain and target domain datasets, which are then input into the training module to complete domain adaptive training. Finally, the local inference module downloads and updates the target detector parameters from the cloud at fixed time points to perform real-time target detection inference on the local video stream.

[0151] The beneficial effects of this invention are as follows:

[0152] 1. The multi-scale feature aggregation and alignment module provided in this application aggregates the output features of different sizes at multiple levels of the backbone network and inputs them into the domain discriminant network to complete adversarial training on the aggregated features of the source domain and the target domain. This can simultaneously avoid the problems of insufficient feature alignment and redundant model parameters, and guide the target detection model to perform preliminary domain adaptation at the image feature level.

[0153] 2. The foreground attention activation module and foreground region weighted alignment module provided in this application weight the aggregated features by obtaining the foreground region activation map from the high-level features of the detection head, and inputting it into the weighted feature domain discriminant network for domain adversarial training. This can enhance the foreground region features in the aggregated features, avoid suboptimal results caused by overtraining the background region, guide the target detection network to focus on the domain adaptation of important foreground regions, and further improve the transfer robustness.

[0154] 3. The instance feature acquisition module and instance feature semantically independent alignment module provided in this application can obtain a single target instance feature vector from aggregated features using source domain labeled bounding boxes and target domain predicted bounding boxes. The vector is then input into a category semantically independent domain discrimination network. Based on category semantically independent loss and adversarial training loss of the same type of features, the semantic separation and independence of different category features and the enhancement of confusion between the same type of feature domains are achieved simultaneously. This avoids the problem of impaired detection performance caused by misalignment of different semantic instance features and guides the semantic independence of cross-domain object detection models at the category level.

[0155] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A foreground activation and semantic independent based cross-domain object detection method, characterized in that, Includes the following steps: Step 1: Obtain the source domain dataset and the target domain dataset, and build a basic single-stage object detection model; Step 2: On the basic single-stage object detection model, a multi-scale feature aggregation and alignment module, a foreground attention activation module, a foreground region weighted alignment module, an instance feature acquisition module, and an instance feature semantically independent alignment module are set up to jointly form a single-stage cross-domain object detection model based on foreground activation and semantic independence. The multi-scale feature aggregation and alignment module on the basic single-stage object detection model specifically includes: Extract output features from different network layers; Aggregate the output features to obtain aggregated features; Construct an aggregated feature domain discrimination network to predict the domain category probability value of aggregated features; Calculate the aggregated feature alignment loss ; A foreground attention activation module and a foreground region weighted alignment module are added to the basic single-stage object detection model, specifically including: Obtain the foreground attention activation map at the detection head; The foreground attention activation map is normalized, filtered, and fused to obtain the foreground attention overlay map. Weighted aggregated features are obtained by using foreground attention overlay map and aggregated features, and then input into the weighted feature domain discriminant network for domain adversarial training. Calculate the weighted alignment loss for weighted aggregated feature adversarial training. ; An instance feature acquisition module and an instance feature semantically independent alignment module are set up on the basic single-stage object detection model, specifically including: Obtain instance characteristics of the source and target domains; Construct a category-semantically independent domain discriminant to predict the category and domain to which instance features belong; Based on the predicted vectors of instance features, calculate the semantically independent loss for different categories. Alignment loss between domains of the same category ; Step 3: Obtain the total loss of the single-stage cross-domain object detection model based on foreground activation and semantic independence, and train it iteratively for multiple rounds. Use the trained single-stage cross-domain object detection model based on foreground activation and semantic independence in the target domain for cross-domain object detection.

2. The method according to claim 1, characterized in that, Step 1, obtaining the source domain dataset and the target domain dataset, specifically includes: Select labeled datasets that are similar to actual application scenarios and have sufficient data volume as source domain datasets, and collect unlabeled data from actual application scenarios to form target domain datasets; The construction of the basic single-stage target detection model specifically includes: The YOLOv3 single-stage object detection model is used as the basic object detector, and Darknet-53 is used as the feature extraction network to extract image features. The image features are fed to three object detection heads (large, medium, and small) through a three-layer branch structure. After processing by the detection heads, the object detection results are obtained. YOLOv3 calculates the detection loss of the basic single-stage object detection model based on the object detection results. .

3. The method according to claim 1, characterized in that, The acquisition of instance features of the source and target domains specifically includes: Collect the bounding box parameters of the source domain annotation, separate the bounding boxes and confidence scores from the detection results obtained from the three detectors corresponding to the target domain training data, and use non-maximum suppression to filter out the bounding box parameters with high confidence scores from the three detectors. Perform ROI Align operation on the bounding box parameters of the source and target domains and the corresponding aggregated features to obtain the instance features of the source and target domains.

4. The method according to claim 3, characterized in that, In step 3, the total loss of the single-stage cross-domain object detection model based on foreground activation and semantic independence is... The loss consists of the detection loss of the basic single-stage object detection model, the loss of the multi-scale feature aggregation and alignment module, the loss of the foreground region weighted alignment module, and the loss of the instance feature semantically independent alignment module, as shown in the following expression: In the formula, This represents the detection loss of the basic single-stage object detection model. This represents the aggregated feature alignment loss. Indicates the weighted alignment loss. This represents semantically independent loss for different categories. This represents the alignment loss between domains of the same category. The balancing coefficient for each loss is set to 0.

1.

1.

5. The method according to claim 4, characterized in that, In step 3, the total loss is used to train the single-stage cross-domain object detection model based on foreground activation and semantic independence in multiple rounds. After training, the domain adaptive model weights are saved and loaded into the basic single-stage object detection model to complete cross-domain object detection in the target domain.

6. A cross-domain target detection system based on foreground activation and semantic independence as described in any one of claims 1-5, characterized in that, It includes a data acquisition module, a training module, and an inference module, among which, The data acquisition module is used to collect data from actual application scenarios and public large datasets related to the detection task. After preprocessing the data, it packages the data into source domain data and target domain data for use by the training module, and at the same time provides the inference module with task data that requires real-time target detection. The training module is used to transfer object detection knowledge from the source domain to the target domain using foreground attention-guided and category semantic independence methods. The inference module is used to receive real-world scene data, perform real-time inference on image data, and output target detection results.

7. The system according to claim 6, characterized in that, The network of the training module is a single-stage cross-domain object detection model based on foreground activation and semantic independence, and the network of the inference module is a basic single-stage object detection model.

Images