Human-intelligent collaborative domain increment target detection method

Abstract

The invention discloses a human-intelligent collaborative domain increment target detection method. According to the invention, an increment strategy and an active learning sampling method are arranged in a flexible increment training loop; reuse and combination of knowledge are realized by using a sparse hybrid expert network, the generalization ability of model increment training is improved, data playback is performed on an old domain sample based on distributed sampling, and knowledge forgetting in the model increment training is relieved; a part of new domain samples with representativeness and diversity are screened through clustering and maximum mean value difference calculation, on one hand, the data size difference of new and old samples is reduced, and on the other hand, the stability and plasticity of the model are balanced by controlling the number proportion of the new and old samples and the round fine granularity of iterative training. According to the method, the adaptability of the team in complex and diversified environments is improved, fine-grained balance of model stability and plasticity is supported, and the method can be applied to a human-intelligent cooperation domain increment target detection scene and has certain practical significance and theoretical value.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer - supported collaborative work. Specifically, it relates to a human - intelligence collaborative domain - incremental object detection method. Background Art

[0002] (1) Domain - incremental Object Detection Technology

[0003] Domain - incremental object detection technology shows extensive application requirements and far - reaching impacts in many fields. For example, in autonomous driving, various in - vehicle cameras detect different road conditions in real - time. The model trained on a sunny dataset will face different weather conditions such as rain, fog, snow, and night. The drift of data distribution leads to a decline in the detection performance of the model, and domain - incremental training of the model is required. Another example is in the task of detecting unlicensed drone flights. The model needs to cope with complex and changing backgrounds, lighting, and shooting angles, which also generates the need for domain - incremental training.

[0004] Domain - incremental object detection technology aims to ensure that the object detection model can maintain and improve its classification and localization capabilities in an environment where the data distribution is constantly changing by designing efficient incremental strategies, so as to achieve sustainable learning of the model. Compared with the category - incremental technology that has been widely studied and applied, the research on domain - incremental object detection technology is still in its early stage, showing great exploration potential and development space. Existing incremental learning methods, such as data replay, data regularization, knowledge distillation, and dynamic network adjustment, provide a certain theoretical basis and technical support for domain - incremental object detection methods. However, domain - incremental object detection technology emphasizes more on how to design effective incremental strategies to reduce the parameter drift problem caused by data distribution drift, thereby enhancing the stability and generalization ability of the model. The latest domain - incremental object detection methods, such as the learning domain bias method LDB and the topology - preserving method TPDIOD, both focus on deeply optimizing the backbone network of the object detection model. LDB freezes all parameters in the backbone network except those related to each domain bias, avoiding parameter drift in the backbone network and learning to adapt to the distribution of the new domain. TPDIOD uses the self - organizing mapping method to select the samples that can best represent the structure of the feature space for data replay, and constrains the drift of the output features of the backbone network to maintain the topological structure of the feature space.

[0005] However, existing domain incremental object detection methods still face some challenges. First, domain incremental training faces the difficulty of fine-grained trade-off between model stability and plasticity. On the one hand, due to issues such as storage limitations, privacy security, and training overhead, the models deployed in practice do not store all training samples for full retraining, resulting in the model being unable to maintain the learned knowledge during incremental training and experiencing catastrophic forgetting. On the other hand, the number of new domain samples and old domain samples is unbalanced. The more new domain data there is, the better the generalization ability of the model, but the degree of forgetting increases. How to balance model stability and plasticity is the focus of current research on incremental learning. Existing domain incremental object detection methods simply use the average precision metric of the model on all tasks as the standard to evaluate stability and plasticity, expecting the detection accuracy of the model to be optimal on all tasks, without considering the constantly changing target needs of human members in practical applications, nor the continuous collaboration ability between humans and models in long-term incremental training tasks. For example, the incremental task goals in the initial stage focus more on plasticity, while those in the later stage focus more on stability, etc. Second, existing methods focus on reducing the parameter drift in the feature space, which effectively improves the continuous learning ability of the model to a certain extent.

[0006] (2) Active learning technology

[0007] In the domain incremental object detection task, new domain training samples usually come from real-time monitored video frames, while the replay samples of the known domain are usually subsets of its training set. The large amount of new domain sample data and the small amount of replay known domain sample data result in an imbalance in the amount of new and old data in incremental training, leading to a decline in the stability of model incremental training. Active Learning technology aims to use various strategies to screen the samples that are most valuable for improving model performance for manual annotation, reduce the new domain samples participating in training, thereby reducing the performance impact caused by data imbalance, and significantly improving the generalization ability of the model under limited data volume. Existing research has explored various model-assisted sampling methods, such as random sampling, diversity-based sampling, model uncertainty-based sampling Entropy and CLUE, committee voting-based sampling TQS, core set-based sampling CoreSet, hybrid sampling AADA and BADGE, etc. The advantages and disadvantages of different active learning strategies depend on factors such as data characteristics and model requirements in specific tasks. Existing active learning methods sample the new samples with the largest amount of information to adapt to the new domain and enhance the model's performance in the new domain. However, when facing the domain incremental object detection scenario, not only the model's performance in the new domain needs to be considered, but also the model's stability on the old domain. Summary of the Invention

[0008] Aiming at the deficiencies of the above-mentioned existing technologies, the present invention proposes a human-intelligence collaborative domain-incremental object detection method, aiming to construct a human-intelligence team composed of human members and an artificial intelligence model, improve the adaptability of the team in complex and diverse environments, and achieve a fine-grained dynamic balance between stability and plasticity.

[0009] To promote the continuous collaboration of the human-intelligence team, the present invention first proposes a human-intelligence collaborative domain-incremental object detection framework, which places the incremental strategy and the active learning sampling method in a flexible incremental training loop, can dynamically adjust and adapt to the requirements of the task, and can perceive the team state in incremental training in real time. This framework not only focuses on optimizing the performance of model incremental training, but also pays more attention to the continuous collaboration ability of the human-intelligence team in the long-term incremental task scenario. Among them, aiming at the problem of poor model stability and generalization caused by data distribution drift in domain-incremental tasks, the present invention proposes a domain-incremental object detection method combining domain-aware mixture of experts and data replay, uses a sparse mixture of experts network to achieve knowledge reuse and combination, improves the generalization ability of model incremental training, and replays the old-domain samples based on distribution sampling to alleviate knowledge forgetting in model incremental training. Aiming at the problem of poor model stability caused by the imbalance between the old and new domain data volumes in domain-incremental tasks, the present invention combines the incremental strategy with the active learning method and proposes an active learning sampling method based on domain distribution difference. By clustering and calculating the maximum mean discrepancy, some representative and diverse new-domain samples are selected. On the one hand, the difference in the data volumes of the old and new samples is reduced, and on the other hand, the stability and plasticity of the model are fine-grained balanced by controlling the number ratio of the old and new samples and the number of iterations of iterative training. The specific technical solutions of the present invention are introduced as follows.

[0010] The present invention provides a human-intelligence collaborative domain-incremental object detection method, which uses a human-intelligence collaborative domain-incremental object detection system to perform domain-incremental object detection; the human-intelligence collaborative domain-incremental object detection system includes a domain-aware mixture of experts object detection network, and the domain-aware mixture of experts object detection network uses ViTDet as the basic model and uses a domain-aware sparse mixture of experts detection network to replace the detection network RCNN of the basic model; the domain-aware sparse mixture of experts detection network includes the following four modules: a domain prompt pool, a domain classifier, a domain-aware gating network, and a sparse mixture of experts network layer; where:

[0011] A learnable domain prompt pool, which is used to attach domain-related information to sample features with different distributions; the domain prompt pool designs a learnable linear layer for each domain, which is used to perform tensor transformation on the input candidate box features, transform them into domain-related prompts, and then use the domain prompt words as attention weights to weight the candidate box features;

[0012] A domain classifier, which receives candidate box features and judges whether they belong to the old domain or the new domain based on these features;

[0013] A domain-aware gating network that receives candidate box features weighted by domain cue attention and outputs an expert score vector for routing samples from different distributions to different expert networks for further feature extraction;

[0014] A sparse mixture-of-experts network layer for replacing four convolutional layers of the base model detection network RCNN; wherein, the mixture-of-experts module layer contains N experts, each expert is a convolutional layer, and the i-th expert network layer is denoted as E i for improving the performance of the detection network in localization and classification tasks.

[0015] In the present invention, the candidate box features input by the domain cue pool and the candidate box features received by the domain classifier are the outputs after the old domain samples and the new domain samples pass through the feature extraction network and the multi-scale feature fusion network in sequence.

[0016] In the present invention, the training process of the human-intelligence collaborative domain incremental object detection system is specifically as follows:

[0017] First, sample the old domain samples to obtain replay data;

[0018] Next, the human member selects the number of samples N and the training budget for each round of active learning sampling, including the training time budget θ t and the efficiency metric θ PT and uses the active learning method to sample and manually annotate the new domain samples;

[0019] Then, use an incremental strategy that combines domain-aware mixture-of-experts and data replay for incremental training;

[0020] Finally, when the current iteration ends, verify on the test set and calculate whether the training budget is exceeded, so as to determine whether further iterative training is required to achieve the team goal.

[0021] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0022] (1) The method includes a human-intelligence collaborative domain incremental object detection framework, and the main body of the framework is a training loop composed of an incremental strategy and an active learning method. The human member can dynamically adjust the team training goal and continuously follow up the training status of the model with the active learning iteration, which helps to build an efficient and reliable human-intelligence collaborative relationship and adapt to the changing team goals in long-term incremental training.

[0023] (2) The method includes a domain incremental object detection method that combines domain-aware mixture-of-experts and data replay, proposes a target detection network based on sparse mixture-of-experts to learn domain-related knowledge, addresses the problem of data distribution drift, and replays a small amount of known domain sample data to alleviate forgetting, improving the stability and plasticity of the model.

[0024] (3) The method includes an active learning sampling method based on domain distribution differences, comprehensively considering the differences and representativeness of new domain samples, selecting samples with smaller differences from the known domain distribution and capable of representing the distribution of new domain samples for incremental training, alleviating the problem of rapid decline in model stability caused by unbalanced data volume, and achieving fine-grained trade-offs between stability and plasticity by controlling the number of iteration rounds of the training loop and the number of new and old domain samples.

[0025] (4) Compared with traditional domain incremental object detection methods, the human-intelligent collaborative domain incremental object detection method proposed by the present invention supports fine-grained trade-offs between model stability and plasticity, can be applied to the scenario of human-intelligent collaborative domain incremental object detection, and has certain practical significance and theoretical value. Description of the Drawings

[0026] Figure 1 is the framework of human-intelligent collaborative domain incremental object detection.

[0027] Figure 2 is the domain incremental object detection method combining domain-aware mixture of experts and data replay.

[0028] Figure 3 is the prototype system page illustration of the active learning sampling module.

[0029] Figure 4 is the prototype system page illustration of manual annotation and annotation document generation.

[0030] Figure 5 is the prototype system page illustration of the incremental training parameter setting of the object detection model.

[0031] Figure 6 is the prototype system page illustration of the incremental training performance monitoring of the object detection model. Detailed Embodiments

[0032] The technical solutions of the present invention will be introduced in detail below in conjunction with the drawings and embodiments.

[0033] The present invention proposes a human - intelligence collaborative domain - incremental object detection framework. Compared with the traditional incremental training process, this framework can achieve a finer - grained trade - off and control of the model's stability and plasticity. Regarding human members and the artificial intelligence model as a human - intelligence team, it focuses on the ability of long - term collaborative incremental training, which can dynamically adjust the team's goals, promote continuous human - intelligence collaboration to meet the dynamic training target requirements in long - term incremental training, and enhance the continuous collaboration ability of the human - intelligence team. At the same time, the present invention discovers that the detection network of the object detection model has a greater impact on the model's stability and plasticity. Therefore, the present invention proposes a domain - incremental object detection method combining domain - aware mixture of experts and data replay. By using the domain - aware mixture of experts object detection network, it improves the knowledge sharing and generalization ability of the model in classification and localization tasks, and by using data replay, it improves the model's stability, achieving stronger stability and generalization ability in the domain - incremental object detection task and solving the problem of data distribution drift in the domain - incremental object detection task. Furthermore, the present invention proposes a new active learning method based on data distribution difference, which comprehensively considers the maximum mean difference between new and old domain data and the diversity and representativeness of the clustering distribution of new domain data, enabling the object detection model to gradually adapt to the new data distribution without catastrophic forgetting. At the same time, combined with multiple rounds of active learning iteration process, it enables human members to dynamically adjust the degree of model incremental training, achieve a fine - grained trade - off between stability and plasticity, and solve the problem of unbalanced data volumes between the new domain and the known domain in the domain - incremental training task. Finally, the present invention can achieve the continuous learning of the object detection model in the domain - incremental task, maintain the model's stability, enhance the model's plasticity, and achieve long - term continuous incremental training.

[0034] 1. Human - intelligence collaborative domain - incremental object detection framework

[0035] Facing the ever - evolving environment, the domain - incremental object detection model aims to accurately detect target objects under the condition of continuously dynamic changes in data distribution. However, in actual deployment applications, the domain - incremental object detection model has different training goals in incremental tasks at different stages. For example, early incremental training tasks require the model to quickly adapt to the new environment and achieve good generalization, while mid - to - late incremental training tasks require the model to maintain existing knowledge and have a higher requirement for model stability. Existing domain - incremental object detection models adopt a standard training process, using a pre - constructed full - volume training dataset for training. The stability and plasticity of the model are closely related to the design of the incremental strategy and the size of the data volume, ignoring the actual needs of human members and making it difficult to achieve a flexible trade - off between stability and plasticity. In addition, human expert knowledge can input some prior knowledge and guidance to the artificial intelligence model, enabling the artificial intelligence model to continuously optimize and improve its ability to solve complex problems and adapt to new environments.

[0036] Therefore, the present invention proposes a human-intelligence collaborative domain-incremental object detection framework, aiming to achieve a fine-grained trade-off between stability and plasticity during the incremental training process by leveraging the expert experience knowledge of human beings, the actual needs of humans, and the data processing and learning capabilities of artificial intelligence. This framework mainly consists of two parts: a domain-incremental object detection method that combines domain-aware mixture of experts and data replay, and an active learning method based on domain distribution differences. The domain-incremental object detection method aims to improve the adaptability of the object detection model by designing an expert network and enhance the stability performance of the model by sampling and replaying old-domain samples. The active learning method is committed to obtaining high-quality new-domain training data and adapting to the dynamic model incremental training requirements of users when the amount of new-domain sample data is excessive. The model selects samples through active learning to reduce the manual annotation cost. Human members provide supervision labels and domain knowledge, balance the data volume of new-domain samples and replayed old-domain samples, and simultaneously set multiple active learning iteration and evaluation processes. Human members can select the number of samples for active learning sampling and annotation and design threshold constraints for the number of incremental training iterations according to the current goals of the team, achieving fine-grained adjustment of the model's stability and plasticity with less annotation cost and training time, which helps human members keep track of the team's training progress.

[0037] The training process guided by the human-intelligence collaborative domain-incremental object detection framework is as follows: First, sample the old-domain samples to obtain replay data; then, human members select the number of samples N for active learning sampling in each round and the training budget, including the training time budget θ t and the efficiency metric θ PT , and use the active learning method to sample and manually annotate the new-domain samples; then, use the incremental strategy that combines domain-aware mixture of experts and data replay for incremental training; finally, at the end of the current iteration, verify and calculate whether the training budget is exceeded to determine whether further iterative training is required to achieve the team's goals.

[0038] The verification of the human-intelligence collaborative domain-incremental object detection framework mainly includes three aspects: training accuracy, the task completion time of human-intelligence collaboration, and training efficiency. Specifically, training accuracy emphasizes the object detection ability of the model under the human-intelligence collaborative domain-incremental framework, and the accuracy metric is AP50 (Average Precision), that is, the area under the precision-recall curve when the intersection over union (IoU) between the predicted box and the ground truth box is 0.5. The task completion time of the human-intelligence team measures the time required for a single incremental task after using the human-intelligence collaborative domain-incremental object detection framework. Let I represent the I-th incremental task, K be the number of iterations, and T I,k be the time required for the k-th iteration. Then the training time for a single iteration is the sampling time T sample , the annotation time T annotation and the incremental training time Ttrain The sum, and the task completion time T I,K Is the sum of the times required for all iterations, expressed as follows:

[0039] T I,k = T sample + T annotation + T train

[0040]

[0041] The training efficiency measures the model accuracy gain per unit time. The accuracy gain PE i,k Is expressed as:

[0042]

[0043] Where R i,j Is the detection accuracy of the model on the test data of the j-th task after i incremental trainings, and the accuracy metric is AP50. The efficiency of the k-th iteration of the i-th incremental task is expressed as PT i,k Which is the ratio of the model training accuracy gain PE i,k And the current iteration training time:

[0044]

[0045] According to the above three verification metrics, human members need to set the training time budget θ t And the efficiency metric θ PT Before the incremental task training to control the number of iterations. The threshold formula can be expressed as follows:

[0046] T I,K < θ t

[0047] PT i,k > θ PT

[0048] 2. Domain-Aware Hybrid Experts and Data Replay Combined Domain Incremental Object Detection Method

[0049] The domain-aware hybrid expert and data replay combined domain incremental object detection method proposed by the present invention can effectively maintain the detection ability of the object detection model for known domain samples, and at the same time can well generalize and learn the knowledge of new domains. Among them, the domain-aware hybrid expert realizes the maintenance of domain common knowledge and the learning of domain-specific knowledge, improves the stability and generalization ability of the model, and data replay helps the model recall and maintain the learned feature space, further improving the stability of the model. The fusion of the two can effectively improve the stability and plasticity of the model. Specifically, the present invention uses the current state-of-the-art ViTDet as the base model, the feature extraction network is Vision Transformer, and the multi-scale feature fusion network includes: a multi-scale adaptation layer (used to upsample and downsample the single-scale output of the feature extraction network to obtain different scale features), a region proposal network RPN (Region Proposal Network, used to extract regions that may contain objects. The RPN used in ViTDet is the RPN of a general two-stage object detection model, which is the same as the RPN structure in the fast rcnn series of models); the output after passing through the feature extraction network and the multi-scale feature fusion network is the region that may contain objects extracted, that is, the candidate box feature.

[0050] Since the reuse of domain common knowledge helps to maintain the stability of the model and the learning of domain-specific knowledge helps to improve the generalization of the model, the present invention proposes a domain-aware sparse hybrid expert detection network to replace the detection network RCNN of the base model, improving the adaptability of the model in classification and localization in multiple scenarios. The domain-aware sparse hybrid expert detection network includes the following four modules: a domain prompt pool, a domain classifier, a domain-aware gating network, and a sparse hybrid expert network layer. The present invention finds that in the initial stage of incremental training, the model tends to map samples belonging to the new domain distribution to the feature space of the known domain, resulting in similar features obtained by samples with different distributions after passing through the backbone network, making it difficult to effectively distinguish and learn. Therefore, the present invention proposes a learnable domain prompt pool for attaching domain-related information to the sample features of different distributions. The domain prompt pool designs a learnable linear layer for each domain, which is used to perform a tensor transformation on the input candidate box features to transform them into domain-related prompts, and then uses the domain prompt words as attention weights to weight the candidate box features, which helps to further separate the feature outputs of different domains, thereby improving the accuracy of the gating network and avoiding routing the features of different domains to the same set of sparse experts. Let Q be the learned domain prompt feature vector, K and V be the candidate box feature vectors flattened after ROI pooling, σ represent the Sigmoid function, and f s be the function for calculating the attention score, then the attention weighting based on the domain prompt can be expressed as:

[0051] Attention(Q,K,V)=σ(f s(ReLU(Q)))·V

[0052] To facilitate the identification of the distribution of samples during the testing and inference processes, so as to select appropriate domain prompts, the present invention proposes a domain classifier that receives the candidate box features output from the multi-scale feature fusion network and determines whether it belongs to a known domain (old domain) or a new domain based on this feature. The domain classifier is trained using the cross-entropy loss function, and the loss function L domainCE is expressed as follows:

[0053]

[0054] where N is the batch size, C is the number of domains, and z i,j is the predicted score of the i-th training sample on the j-th data distribution.

[0055] The domain-aware gating network is a fully connected layer that receives the candidate box features weighted by the domain prompt attention and outputs an expert score vector for routing samples with different distributions to different expert networks for further feature extraction. The candidate box features weighted by the domain prompt are input into the gating network to obtain a score vector g = [g1, g2,..., g N , where N is the number of experts, and g i is the routing score of the i-th expert E i predicted by the gating network. The k experts corresponding to the highest scores in the score vector are selected for feature weighted summation. Finally, the candidate box features y after sparse mixture of experts weighted summation are expressed as:

[0056]

[0057] w i = softmax(g i )

[0058] where S is the index set of the k experts with the highest scores, w i is the normalized score, and x is the candidate box features output by the RPN network. The four convolutional layers of the basic model detection network RCNN are replaced with a sparse mixture of experts network layer. Among them, the mixture of experts module layer contains N experts, and each expert is a convolutional layer. The i-th expert network layer is denoted as E i and is used to improve the performance of the detection network in the localization and classification tasks.

[0059] The total loss function L of the object detection model can be expressed as the sum of the loss L rpn of the region proposal network, the classification loss L cls and the bounding box regression loss L boxReg of the sparse mixture of experts network layer, as well as the domain classifier loss L domainCE :

[0060] L = L rpn + L cls + L boxReg + L domainCE

[0061] Among them, L rpn The loss includes the classification loss and the bounding box regression loss when generating candidate boxes. The classification loss uses the cross-entropy loss function, and the bounding box regression loss uses the Smooth L1 loss function. The loss of the sparse mixture of experts network includes further classification loss L cls for the region of interest and the bounding box regression loss L boxReg , L cls is the cross-entropy loss with the one-hot label of the ground truth, and L boxReg is the Smooth L1 loss function, which is used to adjust the position of the final predicted box.

[0062] In addition, the present invention adopts a data replay method to further improve the stability of the model. During the training process of the new domain incremental object detection task, as the number of training iterations increases, the parameters gradually deviate from the optimal values of the known domain, and the model forgetting intensifies. The data replay method helps the backbone network recall the learned knowledge and avoid overfitting to the feature space and new domain samples. Therefore, the present invention obtains samples reflecting the distribution of the old domain as replay data based on the K-Means++ algorithm. This algorithm first obtains the features of the training samples of the previous domain incremental task and randomly selects a feature point as the initial cluster center. Subsequently, it calculates the distances from the remaining feature points to the initial cluster center, uses the distances as the probabilities for selecting the next cluster center, and the farther the distance, the greater the selection probability. Then, it selects the next cluster center according to the probabilities until all the required number of cluster centers is selected. The old domain samples corresponding to these cluster center features are used as the replay data for the new domain incremental task for training to alleviate forgetting. The core algorithm for sampling old domain samples is as follows:

[0063]

[0064]

[0065] 3. Active learning sampling method based on domain distribution differences

[0066] Traditional incremental training performs full-scale annotation and training on all new domain samples, presenting two challenges: First, there is the issue of imbalance in the data volume between new and old samples. The large volume of new domain data and the small volume of replayed old domain data cause the model to quickly tend towards the new domain data distribution, resulting in catastrophic forgetting on old domain tasks. Second, it is difficult to finely control the model's stability and plasticity. The trade-off between model stability and plasticity is closely related to the data volume of new and old domain samples. The more and better the new domain data, the stronger the model's generalization ability (i.e., enhanced plasticity), but it is also difficult to retain the knowledge of old domain data and it exacerbates forgetting (i.e., reduces stability). Therefore, in the human-machine collaborative domain incremental object detection framework, the present invention proposes an active learning sampling method based on domain distribution differences, aiming to solve the problem of poor model stability caused by data volume imbalance, and at the same time achieve fine-grained dynamic trade-off between model stability and plasticity by controlling the number of new and old domain samples and the number of training rounds. This method comprehensively considers the distributions of known domain samples and the new domain. First, feature extraction is performed on the replayed samples and new domain samples. Then, the features of the new domain samples are clustered, and a certain number of samples are randomly selected from each cluster as the initial set for active learning sampling. This step can enhance the diversity of the active learning sampling samples, ensure that the sampled samples can effectively cover the current distribution of the new domain, and at the same time, random sampling within the cluster ensures the uniformity of the samples, so that the sampling set does not bias towards the densely distributed area, and better reflects the new domain sample distribution under the current model parameters. Then, based on the greedy idea, the sampling is optimized by calculating the maximum mean difference between the selected sample set and the replayed sample set. In each iteration, one sample selected from each cluster is randomly replaced, and after iterating N iter times, the sample set with the smallest maximum mean difference from the replayed sample set is obtained. The samples selected by this method have the smallest difference from the old domain distribution, enabling the model to gradually adapt to the new data distribution and alleviating catastrophic forgetting caused by data distribution drift. Finally, the selected samples are manually annotated and trained together with the replayed samples.

[0067] The active learning sampling method based on domain distribution differences can select diverse and representative samples, enhance the model's adaptability at a lower annotation cost, and make the incremental training of the model smoother. The core algorithm of this active learning sampling process is as follows:

[0068]

[0069] Implementation example:

[0070] This example involves the development of a prototype system for domain incremental object detection tasks, aiming to provide users with a full-process incremental training platform covering the overall process from data sampling, manual annotation, incremental training to model training monitoring and feedback. This prototype system is written in Python and TypeScript languages, and the model is deployed on a Linux server.

[0071] The prototype system is implemented based on a human-machine collaborative domain incremental object detection framework that combines an incremental strategy and an active learning method, and includes the following modules: a data collection and processing module, an active learning module, and a domain incremental object detection module. The data collection and processing module is used to collect image data from actual applications, perform data cleaning and screening, and divide it into a training set and a test set; the active learning module uses the active learning sampling method based on domain distribution difference proposed in the present invention to sample new domain samples, as shown in Figure 3 shown, and submit it to human members for annotation, as shown in Figure 4 shown; the domain incremental object detection module uses the incremental strategy combining domain-aware hybrid experts and data replay proposed in the present invention for incremental learning, and provides parameter configuration and training process monitoring, as shown in Figure 5 and Figure 6 shown respectively.

[0072] Experiments on the domain incremental object detection method based on the incremental strategy and the active learning method are shown in Table 1. Comparative experiments were carried out on the UAV dataset at different locations in MIDGARD and the road dataset with different weather conditions in ACDC. The comparison metric is the mean mAP (mean Average Precision) of the detection accuracy on all known tasks after the i-th incremental training task. The results show that the method of the present invention effectively improves the stability and plasticity of the model in continuous domain incremental object detection tasks.

[0073] Table 1 Comparison experiment results

[0074]

[0075] In addition, 10 users were recruited to use the prototype system developed for the domain incremental object detection task, and semi-structured interviews were conducted. The interview results were sorted out and analyzed. The interview results show that: (1) The prototype system implemented by the present invention supports the trade-off between elastic stability and plasticity indicators by combining incremental learning and active learning, and can cope with and adapt to different task scenarios and requirements; (2) The prototype system implemented by the present invention has a clear process and supports human-machine collaboration and feedback through multiple rounds of annotation and manual correction functions; (3) The prototype system implemented by the present invention has a relatively rich visual interface and supports personalized collaboration.

Claims

1. A domain-incremental target detection method with human-intelligence collaboration, characterized in that: It adopts a human-intelligence-coordinated domain incremental target detection system to perform domain incremental target detection; the human-intelligence-coordinated domain incremental target detection system includes a domain-aware hybrid expert target detection network, which uses ViTDet as a basic model and adopts a domain-aware sparse hybrid expert detection network to replace the detection network RCNN of the basic model; the domain-aware sparse hybrid expert detection network includes the following four modules: a domain prompt pool, a domain classifier, a domain-aware gating network, and a sparse hybrid expert network layer; wherein: A learnable domain hint pool is used to attach domain-related information to sample features of different distributions. The domain hint pool designs a learnable linear layer for each domain to perform tensor transformation on the input candidate box features and convert them into domain-related hints. The domain hint words are then used as attention weights to weight the candidate box features. The domain classifier receives the candidate box features and determines whether it belongs to the old domain or the new domain based on the features; The domain-aware gating network receives the candidate box features weighted by the domain cue attention, inputs the weighted features into the gating network to obtain the expert score vector, which is used to route samples with different distributions to different expert networks for further feature extraction; The sparse hybrid expert network layer is used to replace the four-layer convolution of the basic model detection network RCNN; the hybrid expert module layer contains N experts, each expert is a layer of convolution, and the i-th expert network layer is represented by E i , used to improve the performance of detection networks in positioning and classification tasks.

2. The domain incremental target detection method of human-intelligence collaboration according to claim 1 is characterized in that: The candidate box features input by the domain prompt pool and the candidate box features received by the domain classifier are the outputs of the old domain samples and the new domain samples after passing through the feature extraction network and the multi-scale feature fusion network in turn.

3. The domain-incremental target detection method of human-intelligence collaboration according to claim 1 is characterized in that: The domain-aware gating network is a fully connected layer. The domain-cue weighted candidate box features are input into the gating network to obtain a score vector g = [g1, g2, ..., g N ], N is the number of experts, g i is the i-th expert E predicted by the gating network i The routing score is obtained by selecting the k experts with the highest scores in the score vector for weighted summation of features. Finally, the candidate box feature y after weighted summation of sparse mixed experts is expressed as: w i =softmax(g i ) Where: S is the index set of the k experts with the highest scores, w i is the normalized score, and x is the candidate box feature output by the multi-scale feature fusion network.

4. The domain-incremental target detection method for human-intelligence collaboration according to claim 1, characterized in that: The total loss function L of the target detection model can be expressed as the loss L of the region candidate network rpn , the classification loss L of the sparse mixture of experts network layer cls With the bounding box regression loss L boxReg and the domain classifier loss L domainCE sum: L=L rpn +L cls +L boxReg +L domainCE Among them, L rpn The loss includes the classification loss and bounding box regression loss when generating candidate boxes. The classification loss uses the cross entropy loss function, and the bounding box regression loss uses the Smooth L1 loss function; L cls is the cross entropy loss with the one-hot label of the ground truth, L boxReg is the Smooth L1 loss function, which is used to adjust the final prediction box position; the domain classifier loss L domainCE Use the cross entropy loss function.

5. The domain-incremental target detection method of human-intelligence collaboration according to claim 1, characterized in that: For the old domain samples based on distribution sampling, the K-Means++ algorithm is used to obtain samples reflecting the old domain distribution as replay data for sampling.

6. The domain-incremental target detection method of human-intelligence collaboration according to claim 4 is characterized in that: The active learning sampling method based on domain distribution difference and the domain-aware hybrid expert object detection network are placed in an incremental training loop; the specific steps of the active learning sampling method are as follows: First, feature extraction is performed on the replayed samples and new domain samples; Next, the new domain sample features are clustered, and a certain number of samples are randomly selected in each cluster as the initial active learning sampling set; Then, based on the greedy idea of ​​optimizing sampling, the maximum mean difference between the selected sample set and the replayed sample set is calculated, and one sample selected in each cluster is randomly replaced in each iteration. iter times, find the sample set with the smallest difference in maximum mean from the replay sample set; Finally, the selected samples are manually annotated and trained together with the replayed samples.

7. The domain incremental target detection method of human-intelligence collaboration according to any one of claims 1 to 6, characterized in that: The training process of the domain-incremental object detection system with human-intelligence collaboration is as follows: First, old domain samples are sampled to obtain replay data; Next, the human member selects the number of samples N and the training budget for each round of active learning, including the training time budget θ for the current incremental task t and efficiency index θ PT , and use active learning methods to sample and manually annotate new domain samples; Then, an incremental training strategy combining domain-aware hybrid experts and data replay is used; Finally, the current iteration ends, and the test set is used to verify and calculate whether the training budget is exceeded, so as to determine whether further iterative training is needed to achieve the team's goals.

Images