Day and night cross-domain target detection method and system based on reliable teacher model

Abstract

The application discloses a daytime and nighttime cross-domain target detection method and system based on a reliable teacher model, and the method comprises the following steps: constructing a daytime domain data set with labels and a nighttime domain data set without labels; introducing a reliable pseudo-label filter, a pseudo-label enrichment module, an uncertainty-driven consistency reasoning module and a confidence-weighted unsupervised loss function to construct a daytime and nighttime cross-domain target detection model based on a reliable teacher; and performing cross-domain target detection on the daytime domain data set with labels and the nighttime domain data set without labels to obtain a daytime and nighttime cross-domain target detection result. The application can measure the reliability of negative samples in soft labels by background probability, give lower weights to negative samples with low reliability, reduce the learning of knowledge from the negative samples by the model, and improve the accuracy of the daytime and nighttime cross-domain target detection result. The application can be widely applied to the technical field of target detection as a daytime and nighttime cross-domain target detection method and system based on a reliable teacher model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, and particularly relates to a daytime and nighttime cross-domain target detection method and system based on a reliable teacher model. BACKGROUND

[0002] The performance of the self-training based domain adaptive target detection mainly depends on the quality of the pseudo label, and the existing technology still has a lot of defects in the acquisition and use of the pseudo label, which limits the knowledge transfer from the daytime domain data with labels to the nighttime domain data without labels. Due to the large domain difference between the daytime source image and the nighttime target image, especially in the area with strong nighttime features, such as weak light, glare and uneven illumination, the existing method only obtains the pseudo label through high classification threshold screening, which leads to the problem that a large number of positive samples on the nighttime domain are lost, and the student model cannot learn the relatively complete nighttime domain knowledge. In addition, the pseudo label of the current leading method generally adopts a one-hot encoding form, and the one-hot encoding cannot represent the relationship between classes, which will have an incorrect representation for some classes, leading to the model fitting to the wrong classification and causing the model degradation problem. SUMMARY

[0003] In order to solve the above technical problems, the purpose of the present application is to provide a daytime and nighttime cross-domain target detection method and system based on a reliable teacher model, which can measure the reliability of negative samples in soft labels through background probability and give lower weights to negative samples with low reliability, so as to reduce the knowledge learned by the model and improve the accuracy of daytime and nighttime cross-domain target detection results.

[0004] The first technical solution adopted by the present application is: a daytime and nighttime cross-domain target detection method based on a reliable teacher model, comprising the following steps:

[0005] constructing a daytime domain data set with labels and a nighttime domain data set without labels;

[0006] introducing a reliable pseudo label filter, a pseudo label enrichment module, an uncertainty driven consistency reasoning module and a confidence weighted unsupervised loss function, and constructing a daytime and nighttime cross-domain target detection model based on a reliable teacher;

[0007] The daytime and nighttime cross-domain target detection model based on a reliable teacher performs cross-domain target detection on the daytime domain data set with labels and the nighttime domain data set without labels, and obtains daytime and nighttime cross-domain target detection results.

[0008] Further, the daytime and nighttime cross-domain target detection model based on a reliable teacher performs cross-domain target detection on the daytime domain data set with labels and the nighttime domain data set without labels, and obtains daytime and nighttime cross-domain target detection results, which specifically comprises:

[0009] input the labeled daytime domain data set and the unlabeled nighttime domain data set into the reliable teacher-based daytime and nighttime cross-domain target detection model;

[0010] The region proposal network of the teacher model of the reliable teacher-based daytime and nighttime cross-domain target detection model predicts the unlabeled nighttime domain data set to obtain a pseudo-label candidate set;

[0011] The region proposal network of the student model of the reliable teacher-based daytime and nighttime cross-domain target detection model trains and predicts the labeled daytime domain data set and the unlabeled nighttime domain data set to obtain a region proposal frame;

[0012] The reliable pseudo-label filter of the reliable teacher-based daytime and nighttime cross-domain target detection model filters the pseudo-label candidate set to obtain a pseudo-label set;

[0013] The pseudo-label enrichment module of the reliable teacher-based daytime and nighttime cross-domain target detection model performs nighttime domain foreground target recall on the region proposal frame and the pseudo-label set to obtain a pseudo-frame with a foreground region;

[0014] The RoI head of the teacher model and the student model of the reliable teacher-based daytime and nighttime cross-domain target detection model performs regression classification calculation on the pseudo-frame with the foreground region to obtain a classification calculation result;

[0015] The uncertainty-driven consistency reasoning module of the reliable teacher-based daytime and nighttime cross-domain target detection model introduces a confidence-weighted unsupervised loss function to perform unsupervised loss calculation on the classification calculation result to obtain a daytime and nighttime cross-domain target detection result.

[0016] Further, the unsupervised loss function of the region proposal network of the teacher model and the region proposal network of the student model is specifically as follows:

[0017]

[0018] In the above formula, L cls represents a classification loss function, L reg represents a regression loss function, σ i is used to represent whether the i th anchor point is a positive sample, represents the class prediction and boundary box regression value generated by the student model at the i th anchor point, represents the soft label and boundary box regression value generated by the teacher model at the i th anchor point, represents the unsupervised loss of the region proposal network, N cls represents the number of anchor points, N reg represents the number of positive samples.

[0019] Further, the unsupervised loss function of the RoI head of the teacher model and the RoI head of the student model is specifically as follows:

[0020]

[0021] In the above formula, represents the class probability distribution predicted by the RoI head of the student model for the i-th pseudo box, represents the class probability distribution predicted by the RoI head of the teacher model for the i-th pseudo box, and respectively represent the regression value predicted by the RoI head of the student model and the RoI head of the teacher model for the i-th pseudo box, represents the unsupervised loss of the RoI head, N' cls represents the number of pseudo boxes, represents the classification loss function of the RoI head, N' reg represents the number of foregrounds, σ' i represents whether the i-th pseudo box is a foreground, L reg represents the regression loss function of the RoI head.

[0022] Further, the reliable pseudo label filter of the daytime and nighttime cross-domain target detection model based on a reliable teacher filters the pseudo label candidate set to obtain the pseudo label set, and the specific steps include:

[0023] The pseudo label set is generated by setting a high threshold to filter the pseudo label candidate set, and the expression is:

[0024]

[0025] In the above formula, represents the pseudo label set, δ represents the high threshold, and max(c i ) represents the maximum value of the foreground class probability distribution, represents the pseudo label candidate set, b i and c i respectively represent the coordinates and foreground class probability distribution of the i-th pseudo label in the pseudo label candidate set, represents the number of candidate boxes, r i represents the i-th pseudo label in the pseudo label candidate set.

[0026] Further, the pseudo label enrichment module of the daytime and nighttime cross-domain target detection model based on a reliable teacher performs nighttime domain foreground target recall on the region proposal box and the pseudo label set to obtain a pseudo box with a foreground region, and the specific steps include:

[0027] According to the IoU of the region proposal frame and the pseudo label set, an overlap threshold is set to classify the region proposal frame, wherein the first type of region proposal frame represents the region proposal frame corresponding to the overlap threshold with the IoU higher than the overlap threshold, and the second type of region proposal frame represents the region proposal frame corresponding to the overlap threshold with the IoU lower than the overlap threshold;

[0028] According to the foreground score, the second type of region proposal frame is sorted, and a plurality of second type of region proposal frames with the highest foreground probability are selected to form a second type of region proposal frame set;

[0029] The first type of region proposal frame is combined with the second type of region proposal frame set to obtain a pseudo frame with a foreground region.

[0030] Further, the expression of the confidence weighted unsupervised loss function is specifically as follows:

[0031]

[0032] In the above formula, represents the confidence weighted unsupervised loss function, w i represents the reliability weight, represents the class probability distribution predicted by the RoI head of the student model for the i th pseudo frame, represents the class probability distribution predicted by the RoI head of the teacher model for the i th pseudo frame, L cls represents the classification loss function, s i represents the confidence of the classification of the i th pseudo frame, N' cls represents the number of pseudo frames, s k represents the confidence of the classification of the k th pseudo frame.

[0033] The second technical solution adopted by the present application is: a daytime and nighttime cross-domain target detection system based on a reliable teacher model, comprising:

[0034] The first module is used to construct a labeled daytime domain data set and an unlabeled nighttime domain data set;

[0035] The second module is used to introduce a reliable pseudo label filter, a pseudo label enrichment module, an uncertainty driven consistency reasoning module and a confidence weighted unsupervised loss function to construct a daytime and nighttime cross-domain target detection model based on a reliable teacher;

[0036] The third module is used to perform cross-domain target detection on the labeled daytime domain data set and the unlabeled nighttime domain data set based on the daytime and nighttime cross-domain target detection model based on the reliable teacher to obtain daytime and nighttime cross-domain target detection results.

[0037] The method and system have the advantages that the daytime domain data set with labels and the nighttime domain data set without labels are constructed, a reliable pseudo label filter, a pseudo label enrichment module, an uncertainty driven consistency reasoning module and a confidence weighted unsupervised loss function are further introduced, a daytime nighttime cross domain target detection model based on a reliable teacher is constructed, the reliable pseudo label filter can obtain reliable pseudo labels and reduce the introduction of negative samples, the pseudo label enrichment module can more comprehensively contain the foreground information of the nighttime domain, the uncertainty driven consistency reasoning is designed, the similarity between categories can be represented, the uncertainty of sample labels is effectively represented, the model degradation problem is alleviated, the confidence weighted classification loss function is used, the reliability of negative samples in the soft label is measured through the background probability, the classification loss of the negative samples is weighted through the confidence, the negative samples with low reliability are given low weights, the model learns knowledge from the negative samples, and the problem that the model identifies positive samples as negative samples is alleviated, and finally, the daytime nighttime cross domain target detection model based on the reliable teacher is used for cross domain target detection on the daytime domain data set with labels and the nighttime domain data set without labels, and daytime nighttime cross domain target detection results are obtained. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 is a step flowchart of the daytime nighttime cross domain target detection method based on the reliable teacher model of the present application;

[0039] Figure 2 is a structure block diagram of the daytime nighttime cross domain target detection system based on the reliable teacher model of the present application;

[0040] Figure 3 is a schematic diagram of the daytime nighttime cross domain target detection model based on the reliable teacher provided by the embodiment of the present application;

[0041] Figure 4 is a schematic diagram of the reliable pseudo label filter provided by the embodiment of the present application;

[0042] Figure 5 is a schematic diagram of the pseudo label enrichment module provided by the embodiment of the present application;

[0043] Figure 6 is a schematic diagram of the experimental simulation result provided by the embodiment of the present application. DETAILED DESCRIPTION

[0044] The present application will be further described in detail below in combination with the drawings and specific embodiments. For the step numbers in the following embodiments, only the setting is for the convenience of description, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0045] First of all, it needs to be pointed out that the prior art has the following technical problems:

[0046] 1) Night target detection aims to identify and locate objects in weak light or night images. Target detectors trained on daytime datasets usually have a significant performance drop when applied to night scenes. Early two-stage methods first use an image enhancement module to enhance low-quality images, and then detect the enhanced images, which requires optimization of the enhancement module and the detection module separately. In recent years, methods such as IA-YOLO, PIA, DEDet, and FeatEnhancer explore the joint end-to-end training of image enhancement and target detection modules. Although they have achieved good performance, their training method is supervised training, so their performance is heavily dependent on the annotations of the dataset.

[0047] 2) Unsupervised domain adaptation for object detection (UDA-OD) aims to train a model using labeled source domain data and unlabeled target domain data to improve the model's generalization in the target domain. Current methods include domain conversion, domain alignment, and self-training. Domain conversion methods convert labeled source domain images into target domain style images through image enhancement, and train the model using the converted images to achieve better detection performance in the target domain. However, this approach is not generalizable and requires different enhancement methods for different target domains. Domain alignment methods reduce the detection performance difference between different domains by aligning feature distributions. However, the feature distribution of the target domain may not be uniform, so this method cannot effectively align the features of the source and target domains. Self-training methods use the pre-trained model in the source domain to generate pseudo labels in the target domain to iteratively train the model, and the key challenge is to select high-quality pseudo labels.

[0048] 3) Unsupervised domain adaptation (UDA) in low-light scenarios has attracted widespread attention. Due to the unique challenges posed by low-light conditions, traditional models struggle to cope. Existing research has mostly focused on semantic segmentation and object tracking tasks, with little research on night target detection. 2PCNet proposes a two-stage consistency framework that uses consistency training between student and teacher models and combines image enhancement strategies to address domain differences. However, the model uses pseudo labels selected by a high threshold for consistency training, which can cause some foreground to be missed, affecting the model's detection ability in the night domain.

[0049] Based on this, with reference to Figure 1 , the present application provides a daytime and nighttime cross-domain target detection method based on a reliable teacher model, which comprises the following steps:

[0050] S100, constructing a daytime domain dataset with labels and a nighttime domain dataset without labels;

[0051] In this embodiment, the daytime domain dataset is defined as where this dataset contains images, representing the i-th image, representing the label of the i-th image. The nighttime domain dataset is defined as where contains nighttime images. The UDA aims to train a model using the daytime domain dataset with labels and the nighttime domain dataset without labels to improve the generalization of the model in the nighttime domain.

[0052] S200, introducing a reliable pseudo-label filter, a pseudo-label enrichment module, an uncertainty-driven consistency reasoning module, and a confidence-weighted unsupervised loss function to construct a daytime-nighttime cross-domain target detection model based on a reliable teacher;

[0053] In this embodiment, the main body of the reliable teacher model (Reliable-Teacher, RT) is a teacher-student model, as shown in Figure 3 The model mainly contains four parts: a reliable pseudo-label filter, a pseudo-label enrichment module, an uncertainty-driven consistency reasoning, and a confidence-weighted unsupervised loss. Among them, the reliable pseudo-label filter and the pseudo-label enrichment module constitute a collaborative pseudo-box construction module. To solve the limitations of high threshold pseudo-label screening, the collaborative pseudo-box construction module is used to generate pseudo-boxes containing more positive samples. Specifically, we use the high-confidence pseudo-labels generated by the teacher model to screen the region proposal boxes generated by the region proposal network (Region Proposal Network, RPN) of the student model to obtain a pseudo-box set This part of the pseudo-box set is compared with the high-confidence pseudo-labels More positive samples are included, which enables the model to learn more features of the nighttime domain target. Although more positive samples can be recalled through the above process, the detection performance of the model is still limited by the representation of the pseudo label. Currently, the pseudo label of the state-of-the-art method is generally in the form of one-hot encoding. One-hot encoding cannot represent the relationship between classes, and will have an incorrect representation for some classes, leading to model fitting to the wrong classification and causing model degradation. Therefore, we designed an uncertainty-driven consistency reasoning module, which uses soft labels with class probabilities instead of pseudo labels in the form of one-hot encoding to supervise the training of the student model. Soft labels can represent the similarity between classes and effectively represent the uncertainty of sample labels, thereby alleviating the problem of model overfitting to the wrong pseudo label. In order to alleviate the impact of unreliable negative samples in soft labels, a confidence-weighted unsupervised loss is used in the model. The reliability of soft labels as negative samples is measured by the background probability, which realizes the classification loss of negative samples through confidence weighting. A lower weight is given to negative samples with low reliability, reducing the model's learning of incorrect knowledge from them, and alleviating the problem of identifying positive samples as negative samples.

[0054] S300, based on the reliable teacher, the daytime and nighttime cross-domain target detection model is used for cross-domain target detection on the labeled daytime domain data set and the unlabeled nighttime domain data set, and the daytime and nighttime cross-domain target detection result is obtained.

[0055] First of all, it needs to be explained that in order to improve the generalization of the model in the nighttime domain, the model currently used by a large number of studies is the teacher-student model. The main body of this model is the teacher model and the student model, which have the same structure. The parameters of the teacher model are represented by θ t , and the parameters of the student model are represented by θ s . In the model training process, θ t is updated by the Exponential Moving Average (EMA) method, and θ s is updated by the backpropagation method.

[0056] The teacher model is pre-trained on the daytime domain data set . The pre-trained teacher model can be used to generate pseudo labels on the nighttime domain data set . The student model is trained using the labeled daytime domain data set and the nighttime domain image with pseudo labels. The total loss L is defined as the weighted sum of the daytime domain image supervised loss L s and the nighttime domain unsupervised loss L u , as shown in the following formula:

[0057] L = L s + λL u

[0058] where λ is a weight coefficient.

[0059] In addition, the current self-training based domain adaptive object detection method generally uses a high threshold as a screening condition to screen the candidate pseudo-labels generated by the teacher model. When the classification score of the candidate pseudo-label is greater than the threshold, it can be used as a pseudo-label. Although this method ensures the reliability of the generated pseudo-labels, it will cause many targets with night domain features to be missed. The lack of learning of the student model on this part of the target will limit the detection performance of the model in the night domain.

[0060] To solve this problem, we propose a collaborative pseudo-label construction module, which consists of a reliable pseudo-label filter and a pseudo-label enrichment module. In this way, more targets with night domain features can be recalled to guide the student model to learn more night domain knowledge, thereby improving the detection performance of the student model in the night domain.

[0061] S310, input the labeled daytime domain dataset and the unlabeled nighttime domain dataset to the daytime and nighttime cross-domain object detection model based on the reliable teacher;

[0062] S320, the region proposal network of the teacher model of the daytime and nighttime cross-domain object detection model based on the reliable teacher predicts the unlabeled nighttime domain dataset to obtain a pseudo-label candidate set;

[0063] S330, the region proposal network of the student model of the daytime and nighttime cross-domain object detection model based on the reliable teacher trains and predicts the labeled daytime domain dataset and the unlabeled nighttime domain dataset to obtain a region proposal frame;

[0064] In this embodiment, in the RPN stage, we use classification loss and regression loss to ensure that the prediction results of the same anchor point by the teacher model and the student model are consistent. Since the image enhancement operation does not affect the geometric properties of the image, the anchor points generated by the two models are exactly the same, therefore, minimizing the classification loss and offset regression loss of the student and the teacher in the RPN stage can ensure the consistency between the two models.

[0065] We use and to represent the class prediction and bounding box regression value generated by the student model at the i-th anchor point, respectively, and use and to represent the soft label and bounding box regression value generated by the teacher model at the i-th anchor point, respectively. For each nighttime domain image, the unsupervised loss in the region proposal network can be represented as:

[0066]

[0067] In the above formula, L cls represents a classification loss function, L reg represents a regression loss function, σ i is used to represent whether the i-th anchor point is a positive sample, represents the class prediction and the bounding box regression value generated by the student model at the i-th anchor point, represents the soft label and the bounding box regression value generated by the teacher model at the i-th anchor point, represents the unsupervised loss of the region proposal network, N cls represents the number of anchor points, N reg represents the number of positive samples.

[0068] S340, a reliable pseudo label filter of the daytime-nighttime cross-domain target detection model based on a reliable teacher, filters the pseudo label candidate set to obtain a pseudo label set;

[0069] In this embodiment, in order to obtain reliable pseudo labels and reduce the introduction of negative samples, we filter the pseudo label candidate set generated by the teacher model through a high threshold The process is shown in Figure 4 . Specifically, given an unlabeled nighttime domain dataset D t , the teacher model predicts the dataset to generate a prediction result which is defined as follows:

[0070]

[0071] where b i and c i represent the coordinates and foreground class probability distribution of the i-th pseudo label in the pseudo label candidate set, is the number of candidate boxes. The prediction result is filtered with a high threshold to generate a reliable pseudo label set We can represent the pseudo label set as:

[0072]

[0073] where δ is the threshold value used to filter reliable pseudo labels, and max(c i ) is the maximum value of the foreground class probability distribution. The reliable pseudo labels generated here are used to guide the pseudo label enrichment module to generate pseudo boxes containing rich foregrounds.

[0074] S350, a pseudo label enrichment module of the daytime-nighttime cross-domain target detection model based on a reliable teacher, performs nighttime domain foreground target recall on the region suggestion box and the pseudo label set to obtain a pseudo box with a foreground region;

[0075] In this embodiment, reliable pseudo-labels obtained through high threshold screening often lose a large number of low-confidence positive samples. Therefore, we use the pseudo-label enrichment module to recall some of the discarded nighttime foreground targets. Figure 5 The entire process was demonstrated. First, the student model generated a large number of region suggestion boxes. Next, we filter out the pseudo-boxes. The final filtered pseudo-boxes include suggestion boxes that have a high overlap with the pseudo-labels. Suggestion boxes with low overlap with pseudo-labels but high foreground confidence This set of pseudo-boundaries is compared to the set of pseudo-labels generated by the teacher model. It has richer knowledge of the nighttime domain.

[0076] Specifically, when the same nighttime image is input into the student detector, the region proposal network generates a set of region proposal boxes. We can use the area suggestion box and reliable fake labels IoU, will They are divided into two categories. The set with higher and more reliable pseudo-labels has a higher IoU. This is represented as follows:

[0077]

[0078] Where τ is the IoU threshold, p i yes The i-th suggestion box, y k yes The kth pseudo-tag in the middle.

[0079] We can build and reliably use pseudo-tags Low-overlapping region boxes The details are as follows:

[0080]

[0081] Then we adjust the foreground score accordingly. Sort the suggestion boxes in the table and select the one with the highest foreground probability. The suggestion boxes constitute a set. Where N represents the number of samples used in the region proposal network. The final set of pseudo-boxes. It can be done and The combination yields the following:

[0082]

[0083] This collaborative process incorporates reliable pseudo-labels. and area suggestion box Find the positive samples discarded by the teacher model. The final pseudo box fuses the predictions of the teacher and student models, thus more comprehensively containing the foreground information of the night domain.

[0084] S360, the RoI head of the teacher model and the student model of the reliable teacher-based daytime-nighttime cross-domain target detection model, performs regression classification calculation on the pseudo box with the foreground region, to obtain a classification calculation result;

[0085] In this embodiment, in the cooperative pseudo label construction module, the teacher model and the student model consider IoU and foreground probability, and cooperatively construct a pseudo box set containing rich foreground Then, the pseudo box set is input into the RoI head of the teacher model and the student model to calculate the classification probability. The RoI head helps the student model to better learn new knowledge in the target domain by optimizing the consistency loss between the classification of the teacher model and the classification of the student model.

[0086] For each pseudo box, the teacher model and the student model predict the classification probability and the bounding box regression value. The classification probability predicted by the teacher model is denoted as As a soft label, the unsupervised loss of the final RoI head can be defined as:

[0087]

[0088] In the above formula, denotes the class probability distribution predicted by the RoI head of the student model for the i-th pseudo box, denotes the class probability distribution predicted by the RoI head of the teacher model for the i-th pseudo box, and denote the regression values predicted by the RoI head of the student model and the teacher model for the i-th pseudo box, respectively, denotes the unsupervised loss of the RoI head, and N' cls denotes the number of pseudo boxes, denotes the classification loss function of the RoI head, and N' reg denotes the number of foregrounds, and σ' i denotes whether the i-th pseudo box is a foreground, and L reg denotes the regression loss function of the RoI head.

[0089] wherein σ' i is 1 when the i-th pseudo box is a positive sample, and 0 otherwise; is the class probability distribution predicted by the RoI head of the student model for the i-th pseudo box, and has a dimension of K+1, where K is the number of foreground classes, and the last dimension represents the background class probability; and RoI head of the student model and the teacher model, respectively, represents the regression value predicted by the i-th pseudo box; finally, the loss function is normalized by N cls and N reg are normalized.

[0090] It is worth noting that although our collaborative pseudo label construction module recalls hard-to-identify positive samples, it cannot guarantee that the teacher model provides reliable pseudo labels. Using one-hot encoding to represent pseudo labels for supervision can introduce false pseudo label signals, thereby affecting the performance of the student model in the target domain. In contrast, using soft labels can alleviate overfitting to some extent and learn the correlation between classes, thereby improving the detection performance of the student model in the night domain.

[0091] S370, an uncertainty-driven consistency reasoning module of the daytime-nighttime cross-domain target detection model based on reliable teachers, introduces a confidence-weighted unsupervised loss function to calculate the unsupervised loss of the classification result, and obtains the daytime-nighttime cross-domain target detection result.

[0092] In this embodiment, one-hot encoding can be used to represent the class of the pseudo label, and the index of a certain class is assigned a value of 1, and the remaining indices are assigned a value of 0. However, the teacher model is pre-trained using daytime domain images, which will cause the model to easily produce false predictions on images in the nighttime domain. One-hot encoding cannot represent the relationship between classes, and will have false representations for some classes, causing the model to fit to false pseudo labels and causing model degradation problems.

[0093] To solve the model degradation problem caused by one-hot encoding, we use soft labels to replace one-hot encoding. Soft labels can represent the similarity between classes and effectively represent the uncertainty of sample labels, alleviating the problem of overfitting to false pseudo labels. Specifically, the representation form of the soft label is the class probability distribution predicted by the teacher model.

[0094] In addition, although the collaborative pseudo label construction module can recall many positive samples filtered by a high threshold, it cannot guarantee that the model can always successfully distinguish between positive and negative samples. If the reliability of the negative samples predicted by the teacher model is not measured, the student model will learn too many classification errors in the soft labels, which will prevent the student model from learning some foreground knowledge containing nighttime domain features, and ultimately affect the overall performance.

[0095] To solve this problem, we propose a reliability-based weighting mechanism to generate a reliability weight for each candidate box predicted as a negative sample. These weights are then applied to the classification loss to alleviate the impact of unreliable negative samples in the soft labels.

[0096] For the pseudo box set We can divide it into foreground set and background set The foreground set contains the candidate box whose predicted class probability value is the highest foreground class; the background set contains the candidate box whose predicted class probability value corresponds to the background class. And

[0097] We can define the confidence weighted classification loss function as follows:

[0098]

[0099] Where w i is the reliability weight, mainly used to measure whether the negative sample is reliable, calculated by s i , which is calculated as:

[0100]

[0101] When s = 1, s i takes the value of 1; when s = 0, s i takes the value of the background probability of the pseudo box, which can be obtained from the last dimension of the soft label, and the smaller the value, the lower the possibility of the pseudo box being a negative sample, thereby reducing the fitting of the student model to it.

[0102] The reliability weighted loss function promotes optimization by weighting the background confidence, alleviates the problem of misclassifying positive samples as background by the model, and ultimately improves the detection accuracy.

[0103] Therefore, the embodiment of the application first inputs the night field image into the student model and the teacher model, the teacher model predicts the night field image to generate a pseudo label candidate set The reliable pseudo label filter screens by threshold to obtain a pseudo label set with high classification scores The region proposal network of the student model generates region proposal boxes containing a large number of foreground and background The pseudo label enrichment module is used for processing the pseudo label set and the region proposal box to obtain pseudo boxes that retain a large number of foreground regions The RoI head of the teacher model and the student model is used for regression and classification calculation of the pseudo box The uncertainty driven consistency reasoning module is used for calculating the unsupervised loss of the result of the classification calculation, and improving the recall rate of the model to the positive sample by confidence weighting.

[0104] The uncertainty driven consistency reasoning module is used for calculating the unsupervised loss of the result of the classification calculation, and improving the recall rate of the model to the positive sample by confidence weighting.To solve the problem that high threshold screening pseudo-labels leads to a large number of positive samples being lost, the collaborative pseudo-box construction module (CoPLC) generates pseudo-boxes covering more positive samples through collaboration between the teacher and the student. Specifically, we use high-confidence pseudo-labels generated by the teacher model to screen the region proposal network (RPN) generated by the student model to obtain pseudo-boxes, which contain more positive samples, enabling the model to learn more night domain target features. Although CoPLC can recall more positive samples, the detection performance of the model is still limited by the representation form of the pseudo-labels. In order to alleviate the problem of model classification overfitting to the wrong class, the scheme designs uncertainty-driven consistency reasoning (UnCR), which uses the teacher model to predict the pseudo-boxes generated by the CoPLC module to obtain soft labels with class probability distribution, which are used to supervise the student model training. Soft labels, compared to one-hot encoding, can effectively represent the similarity between classes and the uncertainty of sample labels, effectively alleviating the model degradation problem. In order to further alleviate the problem of positive sample loss, the scheme proposes a reliability-weighted classification loss (RCL), which measures the reliability of negative samples in soft labels through background probability, and realizes the classification loss of negative samples through confidence weighting. For negative samples with low reliability, a lower weight is given to reduce the model's learning from them, thereby alleviating the problem of identifying positive samples as negative samples.

[0105] Further simulation experiments are carried out on the embodiments of the present application:

[0106] To verify the effectiveness of our reliable teacher model in night object detection in autonomous driving and unmanned aerial vehicle reconnaissance scenarios, we conducted experiments on two benchmark datasets and compared our results with eight state-of-the-art (SOTA) methods. Next, we first introduce the experimental dataset and implementation details, and then show detailed experimental results.

[0107] Experimental dataset:

[0108] BDD100K is a large-scale autonomous driving dataset. Refer to 2PCNet, the dataset is divided into two parts, daytime domain and nighttime domain. Only nighttime domain images are used in the test phase. The training set contains 36728 labeled daytime domain images and 32998 unlabeled nighttime domain images, and the test set contains 4707 labeled nighttime domain images. This dataset contains 10 common categories.

[0109] SHIFT is also an autonomous driving dataset. It contains 6 common categories. Refer to 2PCNet, the dataset is divided into two parts, daytime domain and nighttime domain. The dataset is divided into training set and test set. The training set contains 19452 labeled daytime domain images and 8497 unlabeled nighttime domain images, and the test set contains 1200 labeled nighttime domain images.

[0110] Refer to 2PCNet, we use Faster RCNN as the basic detection model, and use ResNet-50 as the feature extractor. Adjust the size of the input image, keep the aspect ratio, and scale the shorter side to 600 pixels. The loss function hyperparameter λ = 1.0, the smoothing coefficient parameter in EMA is set to α = 0.9996, the threshold δ = 0.8, the IoU threshold τ = 0.5, and the number of sampled region proposal boxes N = 512. The hyperparameters involved here strictly follow the state-of-the-art work to ensure comparability. During the pre-training phase of the student model, for the BDD100K dataset, we pre-train the source domain data for 50000 rounds, and for the SHIFT dataset, we pre-train for 20000 rounds. The learning rate used in the entire training phase is 0.02, without using the learning rate decay mechanism, and the model is optimized using the stochastic gradient descent method.

[0111] In the joint training phase, we first copy the parameters of the student model to the teacher model, and then use the nighttime domain data and daytime domain data to train the model. For the BDD100K dataset, train for 40000 rounds, and for the SHIFT dataset, train for 30000 rounds. Data augmentation methods include random horizontal flipping and random picture erasing. All experiments are carried out on two RTX4090 GPUs, with a batch size of 16 source images and 16 target images. The evaluation metric of the model is the average precision of IoU threshold 0.5, which we denote as mAP.

[0112] Comparison with state-of-the-art methods. We compare our reliable teacher model with other eight state-of-the-art methods, including: DAF, UMT, PT, AT, CMT, 2PCNet, MIC and CAT. In addition, we also conduct training of Faster RCNN using only daytime images, whose testing results on nighttime images are referred to as the baseline; training of Faster RCNN using nighttime images with annotations, whose testing results on nighttime images are referred to as the upper bound.

[0113] Result analysis on BDD100K dataset:

[0114] As shown in Table 1, our reliable teacher model achieves the best performance in terms of mAP, reaching 50.6%, which is 9.5% and 4.4% higher than the baseline and the upper bound, respectively. In addition, it is also 3.1% higher than the second highest MIC. Compared with the algorithm 2PCNet specially used for nighttime target detection, our reliable teacher model is 4.2% higher.

[0115] Table 1 Detection performance of reliable teacher model on BDD100K

[0116]

[0117] Result analysis on SHIFT dataset:

[0118] As shown in Table 2, the reliable teacher model achieves the best performance on the SHIFT dataset, with mAP reaching 51.3%. Compared with the second highest mAP of 2PCNet, the reliable teacher model is 2.2% higher.

[0119] Table 2 Detection performance of reliable teacher model on SHIFT

[0120] Method Car Truck Bus Motorcycle Bicycle Pedestrian mAP Baseline 44.5 49.9 53.7 14.3 46.7 40.4 41.6 Upper limit 51.5 56.0 53.6 19.3 52.4 49.7 47.0 DAF 20.4 22.3 30.3 7.9 28.9 18.3 36.5 UMT 47.5 18.4 46.8 16.6 49.2 7.7 31.1 PT 38.8 40.5 54.1 14.8 52.6 43.2 40.7 AT 33.0 54.7 49.5 20.7 52.3 25.8 38.9 CMT 52.7 53.2 55.7 25.1 55.3 52.6 49.1 2PCNet 54.6 54.8 56.6 23.9 54.2 51.4 49.1 MIC 53.1 53.1 56.0 23.2 55.1 51.6 48.7 CAT 48.8 55.1 53.1 22.5 58.2 52.3 48.3 RT 55.1 59.0 58.9 25.1 57.1 52.9 51.3

[0121] Visual result analysis, Figure 6 The detection results of the reliable teacher model and the state-of-the-art methods are compared intuitively. The reliable teacher model realizes more accurate classification through the uncertainty-driven consistency reasoning mechanism (for example, the trucks in the first and second rows). In addition, thanks to the cooperative pseudo-label construction module, the reliable teacher model can successfully locate small objects, making up for the defect of missing positive samples caused by high threshold screening pseudo-labels. The visual results show that the reliable teacher model achieves excellent detection effect on real nighttime datasets.

[0122] Ablation experiment:

[0123] To verify the effectiveness of the key components in the reliable teacher model, we conduct the relevant ablation experiments on the BDD100K dataset. The baseline model adopted here is the basic student-teacher model, and the Faster RCNN is adopted for detection and the image random erasing algorithm is adopted for image enhancement.

[0124] As shown in Table 3, we evaluate three key components through different experiments: collaborative pseudo-label construction module (CoPLC), uncertainty-driven consistency reasoning (UnCR), and reliability-weighted classification loss (RCL). Comparing the results in row A and row B, we find that the model performance does not increase but decreases, and we can analyze that: although CoPLC recalls the positive samples that are difficult to detect, the model performance is limited by the one-hot encoded pseudo-labels, and the wrong pseudo-labels will mislead the student model. Therefore, recalling more positive samples that are difficult to detect will introduce more wrong pseudo-labels, and the model performance will become worse. From row C, we can easily see that when we introduce the UnCR module, the mAP is improved by 1.8% compared with row B, which proves that the UnCR module can prevent the student model from overfitting to the inaccurate prediction of the teacher model. Compared with the baseline model, the mAP is improved by 0.9%, which proves that CoPLC can introduce more night domain foreground information for model learning. By analyzing the results in row D, we can find that when we introduce RCL, the mAP is 1.1% higher than row C, which proves that RCL can reduce the influence of unreliable negative samples.

[0125] Table 3 Influence of different components on the BDD100K dataset

[0126] Method CoPLC UnCR RCL mAP (%) A (reference method) - - - 48.6 B √ - - 47.7 C √ √ - 49.5 D (RT) √ √ √ 50.6

[0127] To further verify the influence of CoPLC, we evaluate the influence of different sampling strategies, and the results are shown in Table 4. When we use the up-sampling method to sample the region proposal with high foreground probability, the detection performance of the model reaches the best, and the mAP reaches 50.6%. When we use the down-sampling method to sample the region proposal with low foreground probability, the mAP decreases significantly, which shows that the sampling method adopted by CoPLC can provide more foreground knowledge. In addition, we also evaluate the random sampling method commonly used in the current frontier method, and its performance is lower than the method proposed.

[0128] Table 4 Influence of different region proposal sampling methods on the BDD100K dataset

[0129]

[0130] Referring Figure 2 , the daytime and nighttime cross-domain target detection system based on a reliable teacher model comprises:

[0131] A first module 201 is configured to construct a daytime domain dataset with labels and a nighttime domain dataset without labels.

[0132] A second module 202 is configured to introduce a reliable pseudo-label filter, a pseudo-label enrichment module, an uncertainty-driven consistency reasoning module and a confidence-weighted unsupervised loss function, and construct a daytime and nighttime cross-domain target detection model based on a reliable teacher.

[0133] A third module 203 is configured to perform cross-domain target detection on the daytime domain dataset with labels and the nighttime domain dataset without labels based on the daytime and nighttime cross-domain target detection model based on a reliable teacher, and obtain a daytime and nighttime cross-domain target detection result.

[0134] The content in the method embodiments is applicable to the system embodiments, the system embodiments specifically implement the same functions as the method embodiments, and achieve the same beneficial effects as the method embodiments.

[0135] The above is a specific description of the preferred embodiments of the present application, but the present application is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application. These equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A day-night cross-domain target detection method based on a reliable teacher model, characterized in that, The method comprises the following steps: constructing a daytime domain data set with labels and a nighttime domain data set without labels; introducing a reliable pseudo-label filter, a pseudo-label enrichment module, an uncertainty-driven consistency reasoning module, and a confidence-weighted unsupervised loss function to construct a daytime and nighttime cross-domain target detection model based on a reliable teacher; inputting the daytime domain data set with labels and the nighttime domain data set without labels into the daytime and nighttime cross-domain target detection model based on the reliable teacher; predicting the nighttime domain data set without labels by using the region proposal network of the teacher model of the daytime and nighttime cross-domain target detection model based on the reliable teacher to obtain a pseudo-label candidate set; training and predicting the daytime domain data set with labels and the nighttime domain data set without labels by using the region proposal network of the student model of the daytime and nighttime cross-domain target detection model based on the reliable teacher to obtain a region proposal frame; screening the pseudo-label candidate set by using the reliable pseudo-label filter of the daytime and nighttime cross-domain target detection model based on the reliable teacher to obtain a pseudo-label set; performing nighttime domain foreground target recall on the region proposal frame and the pseudo-label set by using the pseudo-label enrichment module of the daytime and nighttime cross-domain target detection model based on the reliable teacher to obtain a pseudo frame with a foreground region; performing regression classification calculation on the pseudo frame with the foreground region by using the RoI head of the teacher model and the student model of the daytime and nighttime cross-domain target detection model based on the reliable teacher to obtain a classification calculation result; performing unsupervised loss calculation on the classification calculation result by using the uncertainty-driven consistency reasoning module of the daytime and nighttime cross-domain target detection model based on the reliable teacher and introducing a confidence-weighted unsupervised loss function to obtain a daytime and nighttime cross-domain target detection result. 2.The day and night cross-domain object detection method based on a reliable teacher model according to claim 1, characterized in that, The unsupervised loss function of the region proposal network of the teacher model and the region proposal network of the student model is specifically as follows: In the above formula, represents a classification loss function, represents a regression loss function, is used to represent whether the first anchor point is a positive sample, , represents the class prediction and the bounding box regression value generated by the student model at the first anchor point, , represents the soft label and the bounding box regression value generated by the teacher model at the first anchor point, represents the unsupervised loss of the region proposal network, represents the number of anchor points, represents the number of positive samples. 3.The day and night cross-domain object detection method based on reliable teacher model according to claim 2, characterized in that, The unsupervised loss function of the RoI head of the teacher model and the RoI head of the student model is specifically as follows: In the above formula, the class probability distribution predicted by the RoI head of the student model for the th pseudo box, the class probability distribution predicted by the RoI head of the teacher model for the th pseudo box, and the regression value predicted by the RoI head of the student model and the teacher model for the th pseudo box, respectively, the unsupervised loss of the RoI head, the number of pseudo boxes, the classification loss function of the RoI head, the number of foregrounds, whether the th pseudo box is a foreground, the regression loss function of the RoI head.

4. The day and night cross-domain target detection method based on a reliable teacher model according to claim 3, characterized in that, The step of screening the pseudo-label candidate set by using the reliable pseudo-label filter of the daytime and nighttime cross-domain target detection model based on the reliable teacher to obtain a pseudo-label set specifically comprises: screening the pseudo-label candidate set by setting a high threshold to generate a pseudo-label set, and the expression is as follows: In the above formula, Represents a set of pseudo-tags. Indicates a high threshold. This represents the maximum value of the probability distribution of the foreground category. Represents the candidate set of pseudo-labels. and These represent the first pseudo-label candidate set. The coordinates and foreground category probability distribution of each pseudo-label. Indicates the number of candidate boxes. Indicates the first pseudo-label candidate set A pseudo-label.

5. The day and night cross-domain target detection method based on a reliable teacher model according to claim 4, characterized in that, The step of performing nighttime domain foreground target recall on the region proposal frame and the pseudo-label set by using the pseudo-label enrichment module of the daytime and nighttime cross-domain target detection model based on the reliable teacher to obtain a pseudo frame with a foreground region specifically comprises: classifying the region proposal frame according to the IoU of the region proposal frame and the pseudo-label set, wherein a first type of region proposal frame represents a region proposal frame with an IoU higher than the overlap threshold, and a second type of region proposal frame represents a region proposal frame with an IoU lower than the overlap threshold; sorting the second type of region proposal frame according to the foreground score, and selecting a plurality of second type of region proposal frames with the highest foreground probability to form a second type of region proposal frame set; combining the first type of region proposal frame and the second type of region proposal frame set to obtain a pseudo frame with a foreground region.

6. The day and night cross-domain target detection method based on a reliable teacher model according to claim 5, characterized in that, The expression of the confidence-weighted unsupervised loss function is specifically as follows: In the above formula, This represents the confidence-weighted unsupervised loss function. Represents reliability weight. Represents the RoI head of the student model for the first The class probability distribution obtained from the prediction of each pseudo-box The RoI head of the teacher model represents the first... The class probability distribution obtained from the prediction of each pseudo-box Represents the classification loss function. Indicates the first The confidence score of each pseudo-box classification. Indicates the number of pseudo-boxes. Indicates the first The confidence level of each pseudobox classification.

7. A day-night cross-domain target detection system based on a reliable teacher model, characterized in that, The method comprises the following modules: A first module for constructing a daytime domain dataset with labels and a nighttime domain dataset without labels; A second module for introducing a reliable pseudo-label filter, a pseudo-label enrichment module, an uncertainty-driven consistency reasoning module, and a confidence-weighted unsupervised loss function to construct a daytime and nighttime cross-domain target detection model based on a reliable teacher; A third module for inputting the daytime domain dataset with labels and the nighttime domain dataset without labels into the daytime and nighttime cross-domain target detection model based on the reliable teacher; The region proposal network of the teacher model of the daytime and nighttime cross-domain target detection model based on the reliable teacher predicts the nighttime domain dataset without labels to obtain a pseudo-label candidate set; The region proposal network of the student model of the daytime and nighttime cross-domain target detection model based on the reliable teacher trains and predicts the daytime domain dataset with labels and the nighttime domain dataset without labels to obtain a region proposal frame; The reliable pseudo-label filter of the daytime and nighttime cross-domain target detection model based on the reliable teacher filters the pseudo-label candidate set to obtain a pseudo-label set; The pseudo-label enrichment module of the daytime and nighttime cross-domain target detection model based on the reliable teacher performs nighttime domain foreground target recall on the region proposal frame and the pseudo-label set to obtain a pseudo frame with a foreground region; The RoI head of the teacher model and the student model of the daytime and nighttime cross-domain target detection model based on the reliable teacher performs regression classification calculation on the pseudo frame with the foreground region to obtain a classification calculation result; The uncertainty-driven consistency reasoning module of the daytime and nighttime cross-domain target detection model based on the reliable teacher introduces a confidence-weighted unsupervised loss function to perform unsupervised loss calculation on the classification calculation result to obtain a daytime and nighttime cross-domain target detection result.

Images