Prototype alignment-based passive domain adaptive target detection model generation method and target detection method

Abstract

The application provides a passive field adaptive target detection model generation method and a target detection method based on prototype alignment. In the process of training by using a teacher model and a student model, the features extracted by the student model and the teacher model are further extracted for target features by using a target detection frame, and the first prototype and the second prototype belonging to the same target category are iteratively updated based on the features of the extracted target detection frame and the target category prediction results output by the student model and the teacher model based on the features of the target detection frame. The alignment loss of the prototype is constructed by using the prototype to align the feature extraction results and the prediction results of the student model and the teacher model. In this way, by iteratively updating the prototype, the historical knowledge is incorporated into the training process, thereby avoiding the problem of model training collapse caused by incorrect supervision signals due to pseudo labels, improving the performance of the target detection model obtained by training, and thereby improving the accuracy of the target detection results output by the target detection model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a source-free domain adaptive object detection model generation method and an object detection method based on class prototype alignment. BACKGROUND

[0002] Due to the availability of widely labeled data sets for model training, great progress has been made in object detection technology. In order to reduce the need for manual data labeling, there are unsupervised domain adaptation (UDA) techniques for object detection. UDA usually assumes that source domain and target domain data can be accessed simultaneously for effective adaptation. However, in the real world, source data may not be accessible due to data privacy, unreliable data transmission, or data security reasons. For example, data used to train large language models (LLM) is usually large and valuable, making it unsuitable for open sharing. In order to overcome the problem of unavailability of source domain data, source-free object detection (SFOD) technology has emerged, which relies only on model weights trained in the source domain and unlabeled data in the target domain to achieve model adaptation from the source domain to the target domain, and achieves good target detection results in the target domain.

[0003] Most existing SFOD methods use a mean teacher framework, in which both the teacher and student models are initialized using a pre-trained source domain model, and the student and teacher models are updated using unlabeled data in the target domain. In this framework, the prediction results of the teacher model are filtered and used as pseudo labels to train the student model, and then the parameters of the student model are used to update the teacher model.

[0004] In the prior art, for a mean teacher framework, the selection of pseudo labels depends on the filtering results of the teacher model's predictions, which results in a certain degree of noise in the pseudo labels, thus bringing false supervision signals, leading to model training collapse, and ultimately affecting the accuracy of the target detection results based on the trained model. SUMMARY

[0005] The present application provides a source-free domain adaptive object detection model generation method and an object detection method based on class prototype alignment, to solve the problem of low accuracy of the target detection results of the object detection model in the prior art, and to improve the accuracy of the target detection results of the object detection model.

[0006] The present application provides a source-free domain adaptive object detection model generation method based on class prototype alignment, comprising:

[0007] input the target domain sample image into a first feature extraction module in the teacher model to obtain a first extracted feature, input the target domain sample image into a second feature extraction module in the student model to obtain a second extracted feature, initial parameters of the teacher model and the student model are obtained based on source domain data, and the source domain data includes a source domain sample image and a target detection label corresponding to the source domain sample image;

[0008] extract a first target feature and a second target feature based on a target detection frame in the first extracted feature and the second extracted feature respectively, the target detection frame is obtained based on a result of target detection of the target domain sample image by the teacher model, input the first target feature into a first target category prediction module of the teacher model to obtain a first target category prediction result, and input the second target feature into a second target category prediction module of the student model to obtain a second target category prediction result;

[0009] each first target feature and the first target category prediction result based on a same preset target category iterates a first category prototype of the preset target category, and each second target feature and the second target category prediction result corresponding to the same target preset category iteratively updates a second category prototype of the preset target category;

[0010] obtain an alignment loss between the first category prototype and the second category prototype, determine a training loss based on the alignment loss, update parameters of the student model based on the training loss, update parameters of the teacher model based on the updated parameters of the student model, obtain the teacher model after multiple parameter updates as a target detection model, and the alignment loss reflects the similarity of the first category prototype and the second category prototype and the category prototype knowledge distillation from the teacher model to the student model.

[0011] According to the target domain sample image input into the first feature extraction module in the teacher model provided by the application, before the target domain sample image is input into the first feature extraction module in the teacher model, comprising:

[0012] performing first data enhancement processing on the target domain sample image;

[0013] Before the target domain sample image is input into the second feature extraction module in the student model, comprising:

[0014] performing second data enhancement processing on the target domain sample image;

[0015] The data enhancement intensity of the second data enhancement processing is greater than that of the first data enhancement processing.

[0016] The method comprises the following steps:

[0017] The intersection-over-union between the detection frames output by the teacher model is determined.

[0018] The method comprises the following steps:

[0019] The first-class prototype is iteratively updated based on the first-class feature and the first-class prediction result corresponding to the same target preset class.

[0020] The first-class prototype is iteratively updated based on the first-class feature.

[0021] The first-class prototype is iteratively updated based on the first-class feature.

[0022] The second-class prototype is iteratively updated based on the second-class feature and the second-class prediction result corresponding to the same target preset class.

[0023] The second-class feature is determined based on the normalized second-class prediction result and the second-class feature corresponding to the target preset class in the target domain sample image.

[0024] The first-class prototype is iteratively updated based on the second-class feature.

[0025] The second-class prototype is iteratively updated based on the second-class feature.

[0026] According to the application, a passive field adaptive target detection model generation method based on class prototype alignment is provided.

[0027] A prototype contrast loss is determined based on the similarity between the first class prototype and the second class prototype.

[0028] The first class prototype is input into the teacher model and the student model respectively to obtain first and second class prediction results, and the second class prototype is input into the teacher model and the student model respectively to obtain third and fourth class prediction results.

[0029] A prototype distillation loss is determined based on the distribution difference between the first, second, third and fourth class prediction results.

[0030] The alignment loss is determined based on the prototype contrast loss and the prototype distillation loss.

[0031] The application further provides a target detection method based on the passive field adaptive target detection model generation method based on class prototype alignment.

[0032] A to-be-detected image is obtained and input into the target detection model.

[0033] A target detection result for the to-be-detected image output by the target detection model is obtained.

[0034] The application further provides a passive field adaptive target detection model generation device based on class prototype alignment.

[0035] A feature extraction module is configured to input a target domain sample image into a first feature extraction module of a teacher model to obtain first extracted features, and input the target domain sample image into a second feature extraction module of a student model to obtain second extracted features.

[0036] a category prediction module configured to extract first target features and second target features from the first extracted features and the second extracted features respectively based on target detection boxes, the target detection boxes being obtained based on a result of target detection on the target domain sample image by the teacher model, input the first target features into a first target category prediction module of the teacher model to obtain first target category prediction results, and input the second target features into a second target category prediction module of the student model to obtain second target category prediction results;

[0037] a category prototype updating module configured to update a first category prototype of a same preset target category based on each of the first target features and the first target category prediction results of the same preset target category, and update a second category prototype of the same preset target category based on each of the second target features and the second target category prediction results of the same preset target category;

[0038] a parameter updating module configured to obtain an alignment loss between the first category prototype and the second category prototype, determine a training loss based on the alignment loss, update parameters of the student model based on the training loss, update parameters of the teacher model based on the updated parameters of the student model, and obtain the teacher model after multiple parameter updates as a target detection model, the alignment loss reflecting similarity of the first category prototype and the second category prototype and category prototype knowledge distillation from the teacher model to the student model.

[0039] The application further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor implementing the above-mentioned any one kind of passive field adaptive target detection model generation method based on category prototype alignment when executing the program.

[0040] The application further provides a non-transitory computer readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the above-mentioned any one kind of passive field adaptive target detection model generation method based on category prototype alignment.

[0041] The application further provides a computer program product including a computer program, the computer program being executable by a processor to implement the above-mentioned any one kind of passive field adaptive target detection model generation method based on category prototype alignment.

[0042] The application provides a passive field adaptive target detection model generation method based on class prototype alignment and a target detection method. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0044] Figure 1 FIG. 1 is a flowchart of the passive field adaptive target detection model generation method based on class prototype alignment provided by the application.

[0045] Figure 2 FIG. 2 is a schematic diagram of the overall architecture of the passive field adaptive target detection model generation method based on class prototype alignment provided by the application.

[0046] Figure 3 FIG. 3 is a schematic diagram of model training in the passive field adaptive target detection model generation method based on class prototype alignment provided by the application.

[0047] Figure 4 FIG. 4 is a schematic diagram of model testing in the passive field adaptive target detection model generation method based on class prototype alignment provided by the application.

[0048] Figure 5 FIG. 5 is a structural schematic diagram of the passive field adaptive target detection model generation device based on class prototype alignment provided by the application.

[0049] Figure 6 FIG. 6 is a structural schematic diagram of the electronic device provided by the application. DETAILED DESCRIPTION

[0050] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0051] The present application provides a class prototype alignment-based passive field adaptive target detection model generation method, as shown in the following Figures 1-4 The present application provides a class prototype alignment-based passive field adaptive target detection model generation method, as shown in the following Figure 1 The present application provides a class prototype alignment-based passive field adaptive target detection model generation method, as shown in the following

[0052] S110, inputting the target domain sample image into a first feature extraction module in the teacher model to obtain first extracted features, and inputting the target domain sample image into a second feature extraction module in the student model to obtain second extracted features, the initial parameters of the teacher model and the student model being obtained based on source domain data, the source domain data including source domain sample images and target detection labels corresponding to the source domain sample images;

[0053] S120, extracting first target features and second target features in the first extracted features and the second extracted features based on target detection boxes, the target detection boxes being obtained based on target detection results of the target domain sample image by the teacher model, inputting the first target features into a first target category prediction module of the teacher model to obtain first target category prediction results, and inputting the second target features into a second target category prediction module of the student model to obtain second target category prediction results;

[0054] S130, determining a first class prototype of a preset target category based on each first target feature corresponding to the same preset target category according to the first target category prediction results, and determining a second class prototype of the preset target category based on each second target feature corresponding to the same preset target category according to the second target category prediction results;

[0055] S140, obtaining an alignment loss between the first class prototype and the second class prototype, determining a training loss based on the alignment loss, updating the parameters of the student model based on the training loss, updating the parameters of the teacher model based on the updated parameters of the student model, and obtaining the teacher model after multiple parameter updates as a target detection model, the loss of which reflecting the similarity between the first class prototype and the second class prototype and the class prototype knowledge distillation from the teacher model to the student model.

[0056] The method provided by the application realizes the incorporation of historical knowledge into the training process through iterative updating of the prototype, thereby avoiding the problem of model training collapse caused by incorrect supervision signals brought by pseudo labels, improving the performance of the target detection model obtained by training, and thereby improving the accuracy of the target detection result output by the target detection model.

[0057] The initial parameters of the teacher model and the student model are trained based on labeled source domain data. In the source-free domain adaptive target detection task, the samples for training the model include two parts, namely, source domain labeled samples and target domain unlabeled samples , denoted as , wherein denotes the sample, denotes the corresponding label, including the target frame and the class. denotes the total number of label samples. , wherein denotes the unlabeled sample, denotes the total number of unlabeled samples. The source domain model weight is trained using the labeled samples in the source domain, and then the student model and the teacher model are initialized using the source domain model weight in the target domain, and only the target domain unlabeled samples are used to train the student and teacher models.

[0058] As shown in Figure 3 , in the training process of the student model and the teacher model, the sample data of the target domain are respectively input into the teacher model and the student model, and the parameters of the student model and the teacher model are updated based on the outputs of the teacher model and the student model. After the training is completed, the teacher model is used as a target detection model for target detection on the data of the target domain, as shown in Figure 4 . The training process of the teacher model and the student model is described in detail below.

[0059] As shown in Figure 2 , before the target domain sample images are input into the teacher model and the student model, the target domain sample images can be enhanced, that is, before the target domain sample images are input into the first feature extraction module in the teacher model, the following steps are included:

[0060] performing first data enhancement processing on the target domain sample images;

[0061] Before the target domain sample images are input into the second feature extraction module in the student model, the following steps are included:

[0062] performing second data enhancement processing on the target domain sample images.

[0063] The data augmentation strength of the second data augmentation processing is greater than that of the first data augmentation processing, that is, the target domain sample image is input to the teacher model after weak data augmentation, and the target domain sample image is input to the student model after strong data augmentation.

[0064] Data augmentation generates new training samples by transforming and modifying existing data, thereby enhancing the diversity and quantity of the data set. These transformations can be geometric transformations, color transformations, noise addition, etc. This allows the model to see more diverse data during training, thereby improving the model's generalization ability and robustness.

[0065] For the target domain sample image after data augmentation, it is sequentially input into the student model and the teacher model for processing. At each time the target domain sample image is input into the student model and the teacher model, the class prototype is iteratively updated once based on the output of the student model and the teacher model. Specifically, the student model and the teacher model each have a feature extraction module and a target class prediction module. The feature extraction module is used to extract features and output target detection boxes based on the extracted features. For example, the RPN model architecture shown in Figure 2 , the target class prediction module is used to predict the target classes corresponding to the target detection boxes. For example, the RCNN model architecture shown in Figure 2 and Figure 3 It can be understood that the architecture of the student model and the teacher model can not be limited to the combination of RPN+RCNN shown in Figure 2 and Figure 3 It can also be other neural network model architectures.

[0066] In target detection, target detection boxes with high positioning quality often exist in dense candidate boxes. In the method provided by the present application, a large number of target detection boxes generated by the teacher model are screened, and target detection boxes with high positioning quality are selected, and then the features extracted by the feature extraction module are further extracted based on the selected target detection boxes.

[0067] Specifically, before extracting the first target feature and the second target feature from the first extracted feature and the second extracted feature based on the target detection box, the method comprises:

[0068] Determine the target detection box based on the intersection over union between the detection boxes output by the teacher model for target detection on the target domain sample image.

[0069] The process of selecting a small number of detection boxes generated by the teacher model to obtain a target detection box can be represented by the formula:

[0070] ;

[0071] ;

[0072] wherein, represents a set IOU threshold value, represents the ith detection box, and N represents the total number of detection boxes generated by the teacher model.

[0073] After selecting the target detection boxes with higher positioning quality from the detection boxes, the class prototypes are iteratively updated using the target detection boxes, so as to store the historical knowledge of the teacher model and the student model.

[0074] By screening the target detection boxes with high positioning quality, it is not only beneficial to construct more representative class prototypes, but also beneficial to improve the positioning quality of pseudo labels.

[0075] The first class prototype of the preset target category is iteratively updated based on each first target feature corresponding to the preset target category and the first target category prediction result, and the first class prototype of the preset target category comprises:

[0076] The first target category prediction result corresponding to the preset target category in the current target domain sample image is normalized to obtain a normalized first target category prediction result, and the current first class feature is determined based on the normalized first target category prediction result and the first target feature corresponding to the preset target category in the current target domain sample image.

[0077] The current first class prototype is iteratively updated based on the first class feature.

[0078] The initial first class prototype is the first class feature corresponding to the target and the sample image input to the teacher model.

[0079] The second class prototype of the preset target category is iteratively updated based on each second target feature corresponding to the preset target category and the second target category prediction result, and the second class prototype of the preset target category comprises:

[0080] The second target category prediction result corresponding to the preset target category in the current target domain sample image is normalized to obtain a normalized second target category prediction result, and the current second class feature is determined based on the normalized second target category prediction result and the second target feature corresponding to the preset target category in the current target domain sample image.

[0081] The current first class prototype is iteratively updated based on the second class feature.

[0082] The initial second class prototype is the second class feature corresponding to the target and the sample image input to the teacher model.

[0083] The iterative updating process of the first type of prototype of the teacher model and the second type of prototype of the student model is similar, and the second type of prototype of the student model is taken as an example for specific description.

[0084] After inputting the current target domain sample image into the student model, the second target feature is obtained from the second extraction feature of the feature extraction module of the student model. Corresponding features.

[0085] In a possible implementation, when extracting the second target feature from the second extraction feature, first, the corresponding feature is further extracted from the second extraction feature based on the position of the target detection frame, and then the further extracted feature is subjected to mean pooling, which can be expressed by the formula:

[0086] ;

[0087] Wherein, represents the second target feature, represents strong augmentation of data, represents the feature corresponding to the candidate frame extracted in the feature map of the student model, represents using mean pooling, and the mean pooling can realize dimension reduction of the feature.

[0088] The obtained second target feature is input into the second target category prediction module in the student model to obtain a prediction result, which can be expressed by the formula: , represents the processing of the second target category prediction model, is the second target category prediction result, and the second target category prediction result is a feature vector, and the elements in the feature vector include the probabilities of belonging to different preset target categories.

[0089] The second target category prediction results corresponding to the same category in the second target category prediction results in the target detection frame are subjected to normalization processing to obtain normalized second target category prediction results , which can be expressed by the formula:

[0090] ;

[0091] Wherein, represents the second target category prediction result corresponding to the i-th target detection frame, and M is the number of target detection frames.

[0092] Through the above processing, the normalized second target category prediction results of each preset target category and the corresponding second target features can be obtained.The normalized second target category prediction result of each preset target category and the corresponding second target feature are combined to obtain a normalized second prediction result matrix and a second target feature matrix , take as the second category feature.

[0093] For each target domain sample image, after inputting into the student model, a second category feature can be obtained, taking the first second category feature as the initial value of the second category prototype, and then obtaining a second category feature each time, and updating the second category prototype based on the second category feature each time, the iterative update can adopt the momentum update (EMA) mode, and the iterative update process of the second category prototype can be expressed by the formula:

[0094] ;

[0095] Wherein, t represents the iteration number, is the momentum update coefficient of EMA, indicates the second category prototype updated at the tthiteration.

[0096] For all C+1 preset target categories (including the background category), the teacher model and the student model are used respectively, to obtain the first category prototype and the second category prototype , wherein C is the number of target domain categories, is the category feature dimension of a single category, .

[0097] As can be seen from the foregoing description, the first category prototype and the second category prototype retain the historical knowledge of the target domain sample image in the teacher model and the student model, and the method provided by the application further adds the alignment loss of the first category prototype and the second category prototype to the total loss of model training, uses the category prototype to construct the prototype contrast loss and the prototype distillation loss, to align the representation and the prediction result of the student model and the teacher model, to promote the knowledge transfer from the source domain to the target domain, to overcome the problem of model training collapse caused by the error descent of model parameter gradient due to the noise of pseudo label, and to improve the performance of the model in the target domain.

[0098] Specifically, the alignment loss between the first category prototype and the second category prototype is obtained, including:

[0099] determining a prototype contrast loss based on the similarity between the first category prototype and the second category prototype;

[0100] inputting the first category prototype into the teacher model and the student model respectively to obtain the first category prediction result and the second category prediction result, and inputting the second category prototype into the teacher model and the student model respectively to obtain the third category prediction result and the fourth category prediction result;

[0101] determine the prototype distillation loss based on the distribution difference between each two of the first type of prediction result, the second type of prediction result, the third type of prediction result and the fourth type of prediction result;

[0102] determine the alignment loss based on the prototype contrast loss and the prototype distillation loss.

[0103] The prototype contrast loss between the first type of prototype and the second type of prototype can be determined by evaluating the similarity between each two of the prototypes of the classes through cosine similarity, which can be specifically expressed by the formula:

[0104]

[0105]

[0106] wherein, represents the prototype contrast loss, represents S calculated based on the data of the corresponding i-th class in the first type of prototype and the second type of prototype, represents S calculated based on the data of the corresponding i-th class in the first type of prototype and the data of the corresponding k-th class in the second type of prototype, and T represents a contrast loss temperature parameter used to adjust the bias relationship of the contrast loss function.

[0107] Further, in the method provided by the present application, the prototype distillation loss is added to the alignment loss of the class prototype to promote knowledge distillation from the teacher model to the student model, align the prediction results of different class prototypes, and the specific calculation formula of the prototype distillation loss can be:

[0108]

[0109]

[0110] wherein, , respectively represent the processing of the second target class prediction module in the student model and the first target class prediction module in the teacher model, represents the KL (Kullback-Leibler) divergence between x and y.

[0111] After obtaining the alignment loss between the first type of prototype and the second type of prototype, the alignment loss and the loss based on the pseudo label can be summed to obtain the total training loss, which is used to update the parameters of the student model. That is , represents the total training loss, represents the loss based on the pseudo label.

[0112] ​​​​The loss based on the pseudo label can be obtained by using the existing loss construction method based on the pseudo label in the unsupervised domain adaptive target detection task, for example:

[0113]

[0114] wherein, represents the loss based on the pseudo label obtained for the target domain sample image , and represents the pseudo label, represents the classification loss of the RPN, represents the regression loss of the RPN, represents the classification loss of the ROI (region of interest), represents the regression loss of the ROI.

[0115] After updating the parameters of the student model by the training loss , the parameters of the teacher model are updated based on the updated parameters of the student model. Specifically, the parameters of the student model can be updated based on each batch (a round includes multiple batches, and one batch includes one target domain sample image) in each round, and the parameters of the teacher model are updated using the student model parameters at the end of each round. After training for multiple rounds (for example, 20 rounds), the training is ended.

[0116] The updating of the parameters of the student model can be represented by the formula: , and the updating of the parameters of the teacher model can be represented by the formula: wherein, is the parameter of the student model, is the parameter of the teacher model, is the learning rate, is the EMA update coefficient.

[0117] After the training is ended, the teacher model is used as a target detection model for testing and actual target detection tasks, as shown in Figure 4 .

[0118] Based on the unsupervised domain adaptive target detection model generation method based on class prototype alignment provided by the present application, the present application further provides a target detection method, comprising:

[0119] obtaining a to-be-detected image, and inputting the to-be-detected image into a target detection model;

[0120] obtaining a target detection result for the to-be-detected image output by the target detection model.

[0121] wherein, the target detection model is the teacher model trained in the unsupervised domain adaptive target detection model generation method based on class prototype alignment provided by the present application.​

[0122] The following describes the class prototype alignment-based passive domain adaptive target detection model generation device provided by the present application. The class prototype alignment-based passive domain adaptive target detection model generation device described below can be correspondingly referred to the class prototype alignment-based passive domain adaptive target detection model generation method described above. As shown in Figure 5 The class prototype alignment-based passive domain adaptive target detection model generation device provided by the present application includes:

[0123] The feature extraction module 510 is configured to input the target domain sample image into a first feature extraction module in the teacher model to obtain first extracted features, and input the target domain sample image into a second feature extraction module in the student model to obtain second extracted features. The initial parameters of the teacher model and the student model are obtained based on source domain data. The source domain data includes a source domain sample image and a target detection label corresponding to the source domain sample image.

[0124] The class prediction module 520 is configured to extract first target features and second target features from the first extracted features and the second extracted features based on a target detection frame, respectively. The target detection frame is obtained based on a result of target detection of the target domain sample image by the teacher model. The first target features are input into a first target class prediction module of the teacher model to obtain a first target class prediction result. The second target features are input into a second target class prediction module of the student model to obtain a second target class prediction result.

[0125] The class prototype update module 530 is configured to iteratively update a first class prototype of a preset target class based on each first target feature and the first target class prediction result of the same preset target class, and iteratively update a second class prototype of the preset target class based on each second target feature and the second target class prediction result of the same preset target class.

[0126] The parameter update module 540 is configured to obtain an alignment loss between the first class prototype and the second class prototype, determine a training loss based on the alignment loss, update the parameters of the student model based on the training loss, update the parameters of the teacher model based on the updated parameters of the student model, and obtain the teacher model after multiple parameter updates as a target detection model. The alignment loss reflects the similarity of the first class prototype and the second class prototype and the class prototype knowledge distillation from the teacher model to the student model.

[0127] Figure 6 An example of an entity structure diagram of an electronic device is shown in Figure 6As shown, the electronic device can include a processor 610, a communications interface 620, a memory 630, and a communications bus 640, wherein the processor 610, the communications interface 620, and the memory 630 complete mutual communication through the communications bus 640. The processor 610 can invoke a logic instruction in the memory 630 to execute a class prototype alignment-based passive domain adaptive target detection model generation method and / or a target detection method. The class prototype alignment-based passive domain adaptive target detection model generation method includes: inputting a target domain sample image into a first feature extraction module in a teacher model to obtain first extracted features, inputting the target domain sample image into a second feature extraction module of a student model to obtain second extracted features, initial parameters of the teacher model and the student model being obtained based on source domain data, and the source domain data including source domain sample images and target detection labels corresponding to the source domain sample images; extracting first target features and second target features based on a target detection box in the first extracted features and the second extracted features, respectively, the target detection box being obtained based on a result of target detection of the target domain sample image by the teacher model, inputting the first target features into a first target category prediction module of the teacher model to obtain first target category prediction results, inputting the second target features into a second target category prediction module of the student model to obtain second target category prediction results; iteratively updating a first class prototype of a preset target category based on each first target feature and the first target category prediction result of the same preset target category, and iteratively updating a second class prototype of the preset target category based on each second target feature and the second target category prediction result corresponding to the same target preset category; obtaining an alignment loss between the first class prototype and the second class prototype, determining a training loss based on the alignment loss, updating parameters of the student model based on the training loss, updating parameters of the teacher model based on the updated parameters of the student model, and obtaining the teacher model after multiple parameter updates as a target detection model, wherein the alignment loss reflects similarity of the first class prototype and the second class prototype and class prototype knowledge distillation from the teacher model to the student model. The target detection method includes: obtaining a to-be-detected image, inputting the to-be-detected image into the target detection model, and obtaining a target detection result of the to-be-detected image output by the target detection model, wherein the target detection model is the teacher model after training in the class prototype alignment-based passive domain adaptive target detection model generation method provided by the present application.

[0128] In addition, the logic instructions in the memory 630 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0129] In another aspect, the present application also provides a computer program product comprising a computer program, the computer program being stored in a non-transitory computer-readable storage medium, and the computer program being executable by a processor to enable a computer to perform the class prototype alignment based source-free domain adaptive object detection model generation method and / or the object detection method provided by the above-mentioned methods, the class prototype alignment based source-free domain adaptive object detection model generation method comprising: inputting a target domain sample image into a first feature extraction module in a teacher model to obtain first extracted features, and inputting the target domain sample image into a second feature extraction module in a student model to obtain second extracted features, initial parameters of the teacher model and the student model being obtained based on source domain data, the source domain data comprising source domain sample images and object detection labels corresponding to the source domain sample images; extracting first target features and second target features based on an object detection box in the first extracted features and the second extracted features respectively, the object detection box being obtained based on a result of object detection of the target domain sample image by the teacher model, inputting the first target features into a first target category prediction module of the teacher model to obtain first target category prediction results, and inputting the second target features into a second target category prediction module of the student model to obtain second target category prediction results; iteratively updating a first class prototype of a preset target category based on each first target feature and first target category prediction result of the same preset target category, and iteratively updating a second class prototype of the preset target category based on each second target feature and second target category prediction result of the same preset target category; obtaining an alignment loss between the first class prototype and the second class prototype, determining a training loss based on the alignment loss, updating parameters of the student model based on the training loss, updating parameters of the teacher model based on the updated parameters of the student model, and obtaining the teacher model after multiple parameter updates as an object detection model, the alignment loss reflecting similarity of the first class prototype and the second class prototype and class prototype knowledge distillation from the teacher model to the student model. The object detection method comprises: obtaining an image to be detected, inputting the image to be detected into the object detection model; and obtaining an object detection result of the image to be detected output by the object detection model, wherein the object detection model is the teacher model after training in the class prototype alignment based source-free domain adaptive object detection model generation method provided by the present application.

[0130] In yet another aspect, the present application also provides a non-transitory computer-readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the class prototype alignment based unsupervised domain adaptive object detection model generation method and / or the object detection method provided by the above method, the class prototype alignment based unsupervised domain adaptive object detection model generation method comprising: inputting a target domain sample image into a first feature extraction module in a teacher model to obtain first extracted features, and inputting the target domain sample image into a second feature extraction module in a student model to obtain second extracted features, initial parameters of the teacher model and the student model being obtained based on source domain data, the source domain data comprising source domain sample images and object detection labels corresponding to the source domain sample images; extracting first target features and second target features in the first extracted features and the second extracted features based on an object detection box, the object detection box being obtained based on a result of object detection of the target domain sample image by the teacher model, inputting the first target features into a first target class prediction module of the teacher model to obtain first target class prediction results, and inputting the second target features into a second target class prediction module of the student model to obtain second target class prediction results; iteratively updating a first class prototype of a preset target class based on each first target feature and first target class prediction result of the same preset target class, and iteratively updating a second class prototype of the preset target class based on each second target feature and second target class prediction result of the same preset target class; obtaining an alignment loss between the first class prototype and the second class prototype, determining a training loss based on the alignment loss, updating parameters of the student model based on the training loss, updating parameters of the teacher model based on the updated parameters of the student model, and obtaining the teacher model after multiple parameter updates as an object detection model, the alignment loss reflecting similarity of the first class prototype and the second class prototype and class prototype knowledge distillation from the teacher model to the student model. The object detection method comprises: obtaining an image to be detected, inputting the image to be detected into the object detection model; and obtaining an object detection result of the image to be detected output by the object detection model, wherein the object detection model is the teacher model after training in the class prototype alignment based unsupervised domain adaptive object detection model generation method provided by the present application.

[0131] The device embodiments described above are only schematic and the units shown as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment. Those skilled in the art can understand and implement without creative labor.

[0132] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0133] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and are not limited; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A passive domain adaptive target detection model generation method based on class prototype alignment, characterized in that, The method comprises the following steps: inputting a target domain sample image into a first feature extraction module in a teacher model to obtain a first extracted feature, and inputting the target domain sample image into a second feature extraction module in a student model to obtain a second extracted feature, initial parameters of the teacher model and the student model being obtained based on source domain data, the source domain data comprising a source domain sample image and a target detection label corresponding to the source domain sample image; extracting a first target feature and a second target feature from the first extracted feature and the second extracted feature based on a target detection frame, the target detection frame being obtained based on a result of target detection of the target domain sample image by the teacher model, inputting the first target feature into a first target category prediction module of the teacher model to obtain a first target category prediction result, and inputting the second target feature into a second target category prediction module of the student model to obtain a second target category prediction result; iterating a first category prototype of a preset target category based on each of the first target features and the first target category prediction result of the same preset target category, and updating a second category prototype of the preset target category based on each of the second target features and the second target category prediction result of the same preset target category; obtaining an alignment loss between the first category prototype and the second category prototype, determining a training loss based on the alignment loss, updating parameters of the student model based on the training loss, updating parameters of the teacher model based on the updated parameters of the student model, and obtaining the teacher model after multiple parameter updates as a target detection model, the alignment loss reflecting similarity of the first category prototype and the second category prototype and category prototype knowledge distillation from the teacher model to the student model.

2. The class prototype alignment based passive domain adaptive target detection model generation method according to claim 1, characterized in that, Before the target domain sample image is inputted into the first feature extraction module in the teacher model, the method comprises the following steps: performing first data augmentation processing on the target domain sample image; Before the target domain sample image is inputted into the second feature extraction module in the student model, the method comprises the following steps: performing second data augmentation processing on the target domain sample image; wherein a data augmentation intensity of the second data augmentation processing is greater than that of the first data augmentation processing.

3. The class prototype alignment based passive domain adaptive target detection model generation method according to claim 1, wherein, Before the first target feature and the second target feature are extracted from the first extracted feature and the second extracted feature based on the target detection frame, the method comprises the following steps: determining the target detection frame based on an intersection over union between detection frames output by the teacher model and obtained by performing target detection on the target domain sample image.

4. The class prototype alignment based passive domain adaptive target detection model generation method according to claim 1, wherein, The iterating of the first category prototype of the preset target category based on each of the first target features and the first target category prediction result of the same preset target category comprises the following steps: normalizing the first target class prediction result corresponding to the preset target class in the current target domain sample image, to obtain a normalized first target class prediction result, and determining a current first class feature based on the normalized first target class prediction result and the first target feature corresponding to the preset target class in the current target domain sample image; iteratively updating the first class prototype based on the first class feature; wherein the initial first class prototype is the first class feature corresponding to the target and sample image input into the teacher model; the iteratively updating the second class prototype of the preset target class based on each second target feature and second target class prediction result corresponding to the same preset target class of the target includes: normalizing the second target class prediction result corresponding to the preset target class in the current target domain sample image, to obtain a normalized second target class prediction result, and determining a current second class feature based on the normalized second target class prediction result and the second target feature corresponding to the preset target class in the current target domain sample image; iteratively updating the first class prototype based on the second class feature; wherein the initial second class prototype is the second class feature corresponding to the target and sample image input into the teacher model.

5. The prototype class alignment based passive domain adaptive target detection model generation method according to claim 1, wherein, the obtaining the alignment loss between the first class prototype and the second class prototype includes: determining a prototype comparison loss based on the similarity between the first class prototype and the second class prototype; inputting the first class prototype into the teacher model and the student model respectively, to obtain a first class prediction result and a second class prediction result, and inputting the second class prototype into the teacher model and the student model respectively, to obtain a third class prediction result and a fourth class prediction result; determining a prototype distillation loss based on the distribution difference between each two of the first class prediction result, the second class prediction result, the third class prediction result and the fourth class prediction result; determining the alignment loss based on the prototype comparison loss and the prototype distillation loss.

6. A target detection method based on the prototype class alignment-based passive field adaptive target detection model generation method of any one of claims 1-5, characterized in that, including: obtaining a to-be-detected image, and inputting the to-be-detected image into the target detection model; obtaining a target detection result for the to-be-detected image output by the target detection model.

7. A passive domain adaptive target detection model generation device based on class prototype alignment, characterized in that, The device includes: a feature extraction module configured to input a target domain sample image into a first feature extraction module of a teacher model to obtain a first extracted feature, and input the target domain sample image into a second feature extraction module of a student model to obtain a second extracted feature, wherein initial parameters of the teacher model and the student model are obtained based on source domain data, and the source domain data includes source domain sample images and target detection labels corresponding to the source domain sample images; a category prediction module configured to extract first target features and second target features from the first extracted features and the second extracted features respectively based on target detection boxes, the target detection boxes being obtained based on a result of target detection on the target domain sample image by the teacher model, input the first target features into a first target category prediction module of the teacher model to obtain first target category prediction results, and input the second target features into a second target category prediction module of the student model to obtain second target category prediction results; a category prototype updating module configured to update a first category prototype of a same preset target category based on each of the first target features and the first target category prediction results of the same preset target category, and update a second category prototype of the same preset target category based on each of the second target features and the second target category prediction results of the same preset target category; a parameter updating module configured to obtain an alignment loss between the first category prototype and the second category prototype, determine a training loss based on the alignment loss, update parameters of the student model based on the training loss, update parameters of the teacher model based on the updated parameters of the student model, and obtain the teacher model after multiple parameter updates as a target detection model, the alignment loss reflecting similarities of the first category prototype and the second category prototype and category prototype knowledge distillation from the teacher model to the student model.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor executes the computer program to implement the passive domain adaptive target detection model generation method based on category prototype alignment and / or the target detection method according to any one of claims 1 to 5. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the passive domain adaptive target detection model generation method based on category prototype alignment and / or the target detection method according to any one of claims 1 to 5.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the passive domain adaptive target detection model generation method based on category prototype alignment and / or the target detection method according to any one of claims 1 to 5. The computer program is executed by the processor to implement the passive domain adaptive target detection model generation method based on category prototype alignment and / or the target detection method according to any one of claims 1 to 5.

Images