Target detector training method and apparatus, target detection method and apparatus, and device

Abstract

The present invention provides a target detector training method and apparatus, a target detection method and apparatus, and a device. The training method comprises: inputting a target image in a target data set among a plurality of data sets into a first feature extraction network to obtain a first feature image; inputting the first feature image into a candidate detection head to obtain target scores of candidate targets and a candidate box; determining a target score loss on the basis of the target scores, and determining a candidate box loss on the basis of the candidate box; inputting the candidate box and the first feature image into a second feature extraction network to obtain a second feature image corresponding to the candidate box; inputting the second feature image into a detection head to obtain a unified class score of a detection target and a plurality of bounding boxes; selecting from among the plurality of bounding boxes a target bounding box corresponding to a class in a label space of the target data set; determining a class score loss on the basis of the unified class score; determining a bounding box loss on the basis of the target bounding box; and optimizing a target detector on the basis of the losses.

Description

Target detector training method, target detection method, device and equipment

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application No. 202311113452.6 filed in China on August 31, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present invention relates to the field of computer vision target detection technology, and more particularly to a target detector training method, a target detection method, a device, and equipment. Background Art

[0004] Traditional object detectors are trained and tested on a single dataset and can only detect objects of the labeled categories that appear in a single dataset, which greatly limits their practical applications. In the field of multi-dataset fusion object detection, it is necessary to use multiple datasets to train object detectors so that they can detect objects of the labeled categories that appear in multiple datasets. However, multi-dataset fusion training suffers from the problem of missing dataset annotations. Each dataset only annotates objects of some labeled categories, leaving some unlabeled labeled categories. During object detector training, it is easy to mistakenly treat missed labeled objects as background, making it difficult to guarantee the performance of the trained object detector.

[0005] Summary of the Invention

[0006] Embodiments of the present invention provide a target detector training method, a target detection method, an apparatus, and a device, which are used to solve the problem that in the process of training a target detector by fusing multiple data sets, missed labeled targets are easily mistaken as background, thereby failing to guarantee the performance of the trained target detector.

[0007] In order to solve the above-mentioned technical problems, the present invention is achieved as follows:

[0008] In a first aspect, an embodiment of the present invention provides a method for training an object detector, wherein the object detector includes a first feature extraction network, a candidate detection head, a second feature extraction network, and a detection head, and the method includes:

[0009] Get multiple datasets;

[0010] Inputting a target image in a target data set among the multiple data sets into a first feature extraction network to obtain a first feature image;

[0011] Inputting the first feature image into a candidate detection head to obtain a target score and a candidate box of the candidate target; determining a target score loss of the candidate detection head according to the target score, and determining a candidate box loss of the candidate detection head according to the candidate box;

[0012] Inputting the candidate frame and the first feature image into a second feature extraction network to obtain a second feature image corresponding to the candidate frame;

[0013] Inputting the second feature image into a detection head to obtain a unified category score and multiple detection frames of the detected target; selecting a target detection frame corresponding to a category in the label space of the target dataset from the multiple detection frames; determining a detection head category score loss based on the unified category score; and determining a detection head detection frame loss based on the target detection frame.

[0014] The object detector is optimized according to the candidate detection head target score loss, the candidate detection head candidate box loss, the detection head category score loss and the detection head detection box loss to obtain an optimized object detector.

[0015] Optionally, each of the data sets includes multiple images and labels corresponding to the images, and the labels include categories and detection boxes of the marked objects.

[0016] Optionally, the candidate detection head outputs a plurality of target scores for the candidate target, the number of which is the same as the number of the data sets, each target score corresponding to a data set, indicating the likelihood that the candidate target is a labeled target in the corresponding data set;

[0017] Determining a candidate detection head target score loss according to the target score includes:

[0018] Obtaining a target score of the candidate target corresponding to the target data set;

[0019] The target score loss of the candidate detection head is determined according to the target score of the candidate target corresponding to the target data set.

[0020] Optionally, the unified category score includes multiple category scores, each category score corresponds to a category in a unified label space, and the unified label space records the categories of all labeled objects in the data set.

[0021] Optionally, determining the detection head category score loss according to the unified category score includes:

[0022] Mapping the unified category score to a dataset category score, where the dataset category score is one or more, each of the dataset category scores corresponding to a category in the label space of the target dataset;

[0023] The detection head category score loss is determined according to the dataset category score.

[0024] Optionally, for each candidate target, the detection head outputs C detection boxes, where C is the number of categories in the unified label space, and each detection box corresponds to a category in the unified label space.

[0025] Optionally, for each candidate target, the detection output has K detection boxes, and each of the detection boxes corresponds to one or more categories in the unified label space.

[0026] Optionally, the method further includes:

[0027] Obtain an initial object detector, wherein a detection head of the initial object detector outputs C detection boxes for each candidate object, where C is the number of categories in the unified label space, and each detection box corresponds to a category in the unified label space;

[0028] Clustering the features corresponding to the C categories of the last fully connected layer of the detection head to obtain features of K categories;

[0029] Among them, the categories in the same cluster share a detection box;

[0030] The detection head of the target detector obtained after clustering processing outputs K detection boxes for each candidate target; the target detector obtained after clustering processing serves as the target detector to be trained.

[0031] Optionally, the detection head includes: a first fully connected layer, a Relu layer, a second fully connected layer, a Relu layer and a third fully connected layer connected in sequence.

[0032] In a second aspect, an embodiment of the present invention provides a method for training an object detector, the object detector including a first feature extraction network, a candidate detection head, a second feature extraction network, and a detection head, the method comprising:

[0033] Get multiple datasets;

[0034] The target detector to be trained is copied to obtain a student detector and a teacher detector;

[0035] Inputting a target image in a target data set among the multiple data sets into the teacher detector to obtain a first detection target, and pseudo-labeling the first detection target that has not been manually labeled;

[0036] Input the target image into the first feature extraction network of the student detector to obtain a first feature image; input the first feature image into the candidate detection head of the student detector to obtain a target score and a candidate box of the candidate target; determine the target score loss of the candidate detection head according to the target score, and determine the candidate box loss of the candidate detection head according to the candidate box; input the candidate box and the first feature image into the second feature extraction network of the student detector to obtain a second feature image corresponding to the candidate box; input the second feature image into the detection head of the student detector to obtain a unified category score and multiple detection boxes of the detected target; select a target detection box corresponding to the category in the label space of the target data set from the multiple detection boxes; determine the detection head category score loss and pseudo-labeling loss according to the unified category score; determine the detection head detection box loss according to the target detection box;

[0037] The object detector is optimized according to the candidate detection head target score loss, the candidate detection head candidate box loss, the detection head category score loss, the detection head detection box loss and the pseudo-labeling loss to obtain an optimized object detector.

[0038] Optionally, each of the data sets includes multiple images and labels corresponding to the images, and the labels include categories and detection boxes of the marked objects.

[0039] Optionally, the candidate detection head outputs a plurality of target scores for the candidate target, the number of which is the same as the number of the data sets, each target score corresponding to a data set, indicating the likelihood that the candidate target is a labeled target in the corresponding data set;

[0040] Determining a candidate detection head target score loss according to the target score includes:

[0041] Obtaining a target score of the candidate target corresponding to the target data set;

[0042] The target score loss of the candidate detection head is determined according to the target score of the candidate target corresponding to the target data set.

[0043] Optionally, the unified category score includes multiple category scores, each category score corresponds to a category in a unified label space, and the unified label space records the categories of all labeled objects in the data set.

[0044] Optionally, determining the detection head category score loss and the pseudo-labeling loss according to the unified category score includes:

[0045] Mapping the unified category score to a dataset category score, where the dataset category score is one or more, each of the dataset category scores corresponding to a category in the label space of the target dataset;

[0046] Determining a detection head category score loss based on the category score of the dataset;

[0047] A pseudo labeling loss is determined according to a pseudo category score in addition to the dataset category score in the unified category score.

[0048] Optionally, for each candidate target, the detection head outputs C detection boxes, where C is the number of categories in the unified label space, and each detection box corresponds to a category in the unified label space.

[0049] Optionally, for each candidate target, the detection output has K detection boxes, and each of the detection boxes corresponds to one or more categories in the unified label space.

[0050] Optionally, the method further includes:

[0051] The teacher detector is periodically updated using the student detector.

[0052] Optionally, the periodically using the student detector to update the teacher detector includes:

[0053] When the learning rate of the target detector meets a preset condition, the student detector is used to update the teacher detector.

[0054] Optionally, the preset condition is that the learning rate of the optimized target detector reaches the lowest point of the Cosine function.

[0055] In a third aspect, an embodiment of the present invention provides a target detection method, which is applied to a target detector, wherein the target detector includes a first feature extraction network, a candidate detection head, a second feature extraction network, and a detection head, and the method includes:

[0056] Inputting the image to be detected into the first feature extraction network to obtain a first feature image;

[0057] Inputting the first feature image into a candidate detection head to obtain an object score and a candidate box of the candidate object;

[0058] Inputting the candidate frame and the first feature image into a second feature extraction network to obtain a second feature image corresponding to the candidate frame;

[0059] The second feature image is input into the detection head to obtain a unified category score and a detection frame of the detection target, wherein the unified category score includes multiple category scores, each category score corresponds to a category in a unified label space, and the unified label space records the categories of targets marked in multiple data sets.

[0060] Optionally, inputting the first feature image into a candidate detection head to obtain a target score and a candidate frame of the candidate target includes:

[0061] The candidate detection head outputs multiple target scores for the candidate target, and the target score with the highest score is selected as the target score for the candidate target.

[0062] In a fourth aspect, an embodiment of the present invention provides a training device for an object detector, wherein the object detector includes a first feature extraction network, a candidate detection head, a second feature extraction network, and a detection head. The training device for the object detector includes:

[0063] A first acquisition module is used to acquire multiple data sets;

[0064] A first feature extraction module is configured to input a target image in a target data set among the multiple data sets into a first feature extraction network to obtain a first feature image;

[0065] A candidate target detection module is configured to input the first feature image into a candidate detection head to obtain a target score and a candidate box of the candidate target; determine a target score loss of the candidate detection head based on the target score, and determine a candidate box loss of the candidate detection head based on the candidate box;

[0066] A second feature extraction module is configured to input the candidate frame and the first feature image into a second feature extraction network to obtain a second feature image corresponding to the candidate frame;

[0067] The target detection module is configured to input the second feature image into a detection head to obtain a unified category score and multiple detection frames for the detected target; select a target detection frame corresponding to a category in the label space of the target dataset from the multiple detection frames; determine a detection head category score loss based on the unified category score; and determine a detection head detection frame loss based on the target detection frame;

[0068] The first optimization module is used to optimize the target detector according to the candidate detection head target score loss, the candidate detection head candidate box loss, the detection head category score loss and the detection head detection box loss to obtain an optimized target detector.

[0069] In a fifth aspect, an embodiment of the present invention provides a training device for an object detector, wherein the object detector includes a first feature extraction network, a candidate detection head, a second feature extraction network, and a detection head, and the training device for the object detector includes:

[0070] A first acquisition module is used to acquire multiple data sets;

[0071] The replication module is used to replicate the target detector to be trained to obtain a student detector and a teacher detector;

[0072] a pseudo-labeling module, configured to input a target image in a target data set in the multiple data sets into the teacher detector to obtain a first detection target, and pseudo-label the first detection target that has not been manually labeled;

[0073] A training module is used to input the target image into the first feature extraction network of the student detector to obtain a first feature image; input the first feature image into the candidate detection head of the student detector to obtain a target score and a candidate box of the candidate target; determine the target score loss of the candidate detection head according to the target score, and determine the candidate box loss of the candidate detection head according to the candidate box; input the candidate box and the first feature image into the second feature extraction network of the student detector to obtain a second feature image corresponding to the candidate box; input the second feature image into the detection head of the student detector to obtain a unified category score and multiple detection boxes of the detected target; select a target detection box corresponding to the category in the label space of the target data set from the multiple detection boxes; determine the detection head category score loss and pseudo-labeling loss according to the unified category score; determine the detection head detection box loss according to the target detection box; optimize the target detector according to the candidate detection head target score loss, the candidate detection head candidate box loss, the detection head category score loss, the detection head detection box loss and the pseudo-labeling loss to obtain an optimized target detector.

[0074] In a sixth aspect, an embodiment of the present invention provides a target detection device, which uses a target detector to detect a target, wherein the target detector includes a first feature extraction network, a candidate detection head, a second feature extraction network, and a detection head, and the target detection device includes:

[0075] A first feature extraction module is used to input the image to be detected into a first feature extraction network of the object detector to obtain a first feature image;

[0076] A candidate target detection module is configured to input the first feature image into a candidate detection head to obtain a target score and a candidate frame of the candidate target;

[0077] A second feature extraction module is configured to input the candidate frame and the first feature image into a second feature extraction network to obtain a second feature image corresponding to the candidate frame;

[0078] The target detection module is used to input the second feature image into the detection head to obtain a unified category score and a detection frame of the detected target, wherein the unified category score includes multiple category scores, each category score corresponds to a category in a unified label space, and the unified label space records the categories of targets marked in multiple data sets.

[0079] In the seventh aspect, an embodiment of the present invention provides an electronic device, comprising: a processor, a memory, and a program stored in the memory and runnable on the processor, wherein when the program is executed by the processor, the steps of the target detector training method as described in the first or second aspect above are implemented, or, when the program is executed by the processor, the steps of the target detection method as described in the third aspect above are implemented.

[0080] In an eighth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the computer program implements the steps of the target detector training method as described in the first or second aspect above; or, when the computer program is executed by a processor, the computer program implements the steps of the target detection method as described in the third aspect above.

[0081] In an embodiment of the present invention, the candidate detection head is first used to roughly detect the candidate frame, and then the detection head is used to finely detect the detection frame and unified category score of the detection target, that is, two-stage target detection is adopted, which can improve the detection accuracy of the trained target detector. In addition, the number of categories in multi-dataset fusion training is usually large, and it is difficult to achieve precise positioning of multiple category detection frames using a single detection frame regression module. In an embodiment of the present invention, by designing a multi-detection frame detection head, precise positioning of multiple category detection frames can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0083] FIG1 is a schematic diagram of a flow chart of a method for training an object detector according to an embodiment of the present invention;

[0084] FIG2 is a schematic diagram of a two-stage target detection method according to an embodiment of the present invention;

[0085] FIG3 is a schematic diagram of the architecture of an object detector according to an embodiment of the present invention;

[0086] FIG4 is a schematic structural diagram of a detection head according to an embodiment of the present invention;

[0087] FIG5 is a schematic flow chart of a method for training an object detector according to another embodiment of the present invention;

[0088] FIG6 is a schematic diagram of a training method for an online pseudo-labeled object detector according to an embodiment of the present invention;

[0089] FIG7 is a schematic flow chart of a target detection method according to an embodiment of the present invention;

[0090] FIG8 is a schematic structural diagram of a training device for an object detector according to an embodiment of the present invention;

[0091] FIG9 is a schematic structural diagram of a training device for an object detector according to another embodiment of the present invention;

[0092] FIG10 is a schematic structural diagram of a target detection device according to an embodiment of the present invention;

[0093] FIG11 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0094] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0095] Referring to Figures 1, 2, and 3, an embodiment of the present invention provides a method for training an object detector, wherein the object detector includes a first feature extraction network, a candidate detection head, a second feature extraction network, and a detection head. The method includes:

[0096] Step 11: Get multiple datasets.

[0097] In an embodiment of the present invention, each of the data sets includes multiple images and labels corresponding to the images, and the labels include categories and detection boxes of the marked objects.

[0098] In this embodiment of the present invention, the categories of labeled objects in different datasets can be completely different or partially overlapping. For example, the categories of labeled objects in dataset D1 include: people, cats, dogs, trees, and houses, while the categories of labeled objects in dataset D2 include: people, trees, flowers, and birds. There are overlapping categories between datasets D1 and D2: people and trees.

[0099] The detection frame in the label is data indicating the position and / or size of the detection frame, and may include the center coordinates (x, y) of the detection frame, as well as the width w and height h of the detection frame. In some embodiments, it may also include the coordinates of the four endpoints of the detection frame.

[0100] In the embodiments shown in FIG. 2 and FIG. 3 , the number of data sets is 3. Of course, the number of data sets is not limited thereto.

[0101] Step 12: inputting a target image in a target data set among the multiple data sets into a first feature extraction network to obtain a first feature image;

[0102] Optionally, the target dataset may be any one of the multiple datasets. The target detector may be trained using images from the multiple datasets in sequence.

[0103] The first feature extraction network can be a backbone network for feature extraction, which can be a CNN backbone network such as ResNet, MobileNet, or a Transformer backbone network similar to SWIN. The backbone network extracts common features of the image.

[0104] Step 13: Input the first feature image into the candidate detection head to obtain the target score and candidate box of the candidate target; determine the target score loss of the candidate detection head according to the target score, and determine the candidate box loss of the candidate detection head according to the candidate box;

[0105] In the embodiment of the present invention, the candidate detection head is used to detect candidate frames of candidate targets that may be real targets and output the target score of the candidate targets and the position information of the candidate frames, without determining the category of the candidate targets.

[0106] The target score represents the possibility that the candidate target is the true target.

[0107] In the embodiment of the present invention, the loss of the candidate detection head includes: the target score loss of the candidate detection head and the candidate box loss of the candidate detection head.

[0108] The target score loss of the candidate detection head is determined according to the target score, which can be specifically: the target score loss of the candidate detection head can be calculated according to the target score of the candidate target and the labeled target in the label corresponding to the target image.

[0109] The candidate detection head candidate frame loss is determined according to the candidate frame, and specifically can be: calculating the candidate detection head candidate frame loss according to the candidate frame of the candidate target and the real detection frame in the label corresponding to the target image.

[0110] When calculating the loss of the candidate detection head, corresponding weights can be assigned to the candidate detection head object score loss and the candidate detection head candidate box loss respectively, and then the sum of the two after weighting is calculated. For example, a weight q (0 < q < 1) is assigned to the candidate detection head object score loss, and a weight 1 - q is assigned to the candidate detection head candidate box loss. At this time, the loss of the candidate detection head can be: q * candidate detection head object score loss + (1 - q) * candidate detection head candidate box loss.

[0111] Step 14: Input the candidate box and the first feature image into the second feature extraction network to obtain the second feature image corresponding to the candidate box;

[0112] In the embodiment of the present invention, the second feature extraction network (which can also be called ROIPooling) is used to extract the features of the candidate target. Specifically, it can be to intercept the screenshot corresponding to the candidate box in the first feature image by using the candidate box to obtain the second feature image. Optionally, the second feature extraction network can use the ROIPooling module of FasterRCNN to extract the features of the candidate target.

[0113] Step 15: Input the second feature image into the detection head to obtain the unified category score of the detection target and multiple detection boxes; select the target detection box corresponding to the category in the label space of the target dataset from the multiple detection boxes; determine the detection head category score loss according to the unified category score; determine the detection head detection box loss according to the target detection box;

[0114] In the embodiment of the present invention, the unified category score includes multiple category scores, and each category score corresponds to a category in the unified label space, and the unified label space records all the categories of the labeled targets in the dataset.

[0115] In the embodiment of the present invention, assume that the number of datasets used for training the target detector is N, and the N datasets are respectively denoted as D1, D2, …, D N , and the label spaces corresponding to the N datasets are respectively denoted as L1, L2, …, L N , and the label space is used to record the labels of the datasets. The label spaces of the N datasets are merged to obtain a unified label space. For the categories in the labels, the merging can be to merge the same categories in the datasets and keep the different categories. The unified label space can be denoted as L, and each label space L i of each dataset is a subset of L, and each label space L i of each dataset has a mapping correspondence with the categories in the unified label space L, and L iA certain category in [it] can always be mapped to the corresponding category in L. For example, assume there are three datasets. The label space of dataset D1 includes 5 categories, the label space of dataset D2 includes 6 categories, and the label space of dataset D3 includes 7 categories. The categories in the unified label space can be 15 (the same categories in the 3 datasets are merged). For example, the mapping correspondence between the unified label space L and the label space L i can be, for example: The 1st, 2nd, 4th, 6th, and 10th categories in the unified label space L correspond one-to-one to the 5 categories in dataset D1.

[0116] In an embodiment of the present invention, the detection head can be obtained by improving on the basis of the Faster RCNN detection head.

[0117] In an embodiment of the present invention, the loss of the detection head includes: the detection head category score loss and the detection head detection box loss.

[0118] In an embodiment of the present invention, optionally, determining the detection head category score loss according to the unified category score may include: calculating the detection head category score loss according to the unified category score of the detection target and the true category of the labeled target in the target image.

[0119] In an embodiment of the present invention, optionally, determining the detection head detection box loss according to the target detection box may include: calculating the detection head detection box loss according to the target detection box and the detection box of the labeled target in the target image.

[0120] When calculating the loss of the detection head, corresponding weights can be assigned to the detection head category score loss and the detection head detection box loss respectively, and then the sum of the two after weighting is calculated. For example, a weight p (0 < p < 1) is assigned to the detection head category score loss, and a weight 1 - p is assigned to the detection head detection box loss. At this time, the loss of the detection head can be: p * detection head category score loss + (1 - p) * detection head detection box loss.

[0121] In an embodiment of the present invention, when calculating the candidate detection head detection box loss, the Soft L1 loss function can be used. Of course, it is not limited thereto, and other types of loss functions can also be used.

[0122] Step 16: Optimize the target detector according to the candidate detection head target score loss, the candidate detection head candidate box loss, the detection head category score loss, and the detection head detection box loss, to obtain an optimized target detector.

[0123] In this embodiment of the present invention, the candidate detection head object score loss and the candidate detection head candidate box loss can also be referred to as the candidate detection head loss, and the detection head category score loss and the detection head detection box loss can also be referred to as the detection head loss. The total loss of the object detector can be the sum of the candidate detection head loss and the detection head loss, or a weighted sum of the two.

[0124] In an embodiment of the present invention, images in multiple data sets can be input into the target detector in sequence to train the target detector. For each input training image, steps 12 to 16 are executed until the training is completed. The condition for the end of training can be that all images in the data sets have participated in the training, or that the trained target detector meets the set conditions, etc.

[0125] In an embodiment of the present invention, the candidate detection head is first used to roughly detect the candidate frame, and then the detection head is used to finely detect the detection frame and unified category score of the detection target, that is, two-stage target detection is adopted, which can improve the detection accuracy of the trained target detector. In addition, the number of categories in multi-dataset fusion training is usually large, and it is difficult to achieve precise positioning of multiple category detection frames using a single detection frame regression module. In an embodiment of the present invention, by designing a multi-detection frame detection head, precise positioning of multiple category detection frames can be achieved.

[0126] In an embodiment of the present invention, optionally, referring to FIG3 , the candidate detection head outputs a plurality of target scores for the candidate target, the number of which is the same as the number of the data sets, and each target score corresponds to a data set, indicating the likelihood that the candidate target is a labeled target in the corresponding data set. In this case, in step 13 above, determining the candidate detection head target score loss based on the target score includes:

[0127] Step 131: Obtain the target score of the candidate target corresponding to the target data set;

[0128] Step 132: Determine the target score loss of the candidate detection head according to the target score of the candidate target corresponding to the target data set.

[0129] Specifically, the target score loss of the candidate detection head may be determined according to the target score of the candidate target corresponding to the target data set and the labeled target in the target image.

[0130] Taking Figure 3 as an example, the candidate detection head outputs three target scores O1, O2, and O3, corresponding to data sets D1, D2, and D3, respectively. Assuming that the target image currently input to the target detector belongs to data set D1, if the target score that does not correspond to the data set D1 among the three target scores is used to calculate the candidate detection head target score loss with the labeled targets of the target image in data set D1, it will inevitably be inaccurate and the missed labeled targets in data set D1 will be mistakenly regarded as background. In an embodiment of the present invention, the candidate detection head target score loss is calculated based on the target score O1 corresponding to data set D1 and the labeled targets of the target image in data set D1, which can avoid mistaking missed labeled targets in the target data set for background.

[0131] In the embodiment of the present invention, optionally, determining the detection head category score loss according to the unified category score includes:

[0132] Step 151: Mapping the unified category score to a dataset category score, where the dataset category score is one or more, and each dataset category score corresponds to a category in the label space of the target dataset;

[0133] Step 152: Determine the detection head category score loss according to the dataset category score.

[0134] The following example illustrates how to map the unified category score to the dataset category score. Assuming that there are 15 categories in the unified label space L, the unified category score that the detection head can output for the detected target includes 15 scores, each corresponding to the 15 categories in the unified label space L. In other words, each category in the unified label space L is scored. Since the target dataset where the target image currently input to the target detector is located only has some categories labeled, assuming that the target image currently input to the target detector belongs to dataset D1, and only 5 categories of targets are labeled in dataset D1, if the 15 category scores and the labels in dataset D1 are used to calculate the loss of the detection head category score, it will inevitably be inaccurate, and the unlabeled targets in dataset D1 will be mistaken for background. To overcome this problem, in an embodiment of the present invention, the unified category score is mapped to the dataset score corresponding to the dataset D1. For example, the mapping method is as follows: assuming that the 1st, 2nd, 4th, 6th, and 10th categories in the unified label space L correspond one-to-one to the 5 categories in the dataset D1, the 1st, 2nd, 4th, 6th, and 10th scores in the unified category score are taken out as the dataset category score of the dataset D1, and the detection head category score loss is calculated based on the dataset category score of the dataset D1 and the labels in the dataset D1 to avoid mistaking the missed labeled targets in the target dataset as background.

[0135] Since the unified category score includes the scores of categories corresponding to all data sets, and the target data set where the current training image is located only labels some categories, if the unified category score is used to calculate the detection head category score loss, it will inevitably be inaccurate, and it is easy to mistakenly regard the missed labeled targets in the target data set as the background. In the present invention, the unified category score is mapped to the data set score of the target data set where the current training image is located, and the detection head category score loss is calculated based on the data set score. This can avoid mistaking the missed labeled targets in the target data set for the background, further improve the detection accuracy of the trained target detector, and improve the performance of the target detector.

[0136] In some embodiments of the present invention, optionally, for each candidate object, the detection head outputs C detection boxes, where C is the number of categories in the unified label space, and each detection box corresponds to a category in the unified label space. That is, each category in the unified label space has a dedicated detection box output.

[0137] In some embodiments of the present invention, optionally, for each candidate target, the detection output has K detection frames, and each detection frame corresponds to one or more categories in the unified label space. That is to say, not every category in the unified label space corresponds to a detection frame output, and some different categories can share a detection frame, thereby reducing the number of detection frames output by the detection head. For example, the image features of categories such as cows, horses, puppies, and kittens are relatively similar, and the detection frame annotation boundaries are also relatively similar, so they can correspond to the same detection frame. On the contrary, categories with large differences in detection frame annotation boundaries are not suitable for sharing a detection frame.

[0138] The following describes how to determine the correspondence between detection frames and categories, that is, how to determine which categories can share a detection frame.

[0139] In an embodiment of the present invention, optionally, the object detector training method further includes:

[0140] Step 01: Obtain an initial object detector. The detection head of the initial object detector outputs C detection boxes for each candidate object, where C is the number of categories in the unified label space, and each detection box corresponds to a category in the unified label space.

[0141] Step 02: Cluster the features corresponding to the C categories of the last fully connected layer of the detection head to obtain features of K categories, where categories in the same cluster share a detection box; the detection head of the target detector obtained after clustering processing outputs K detection boxes for each candidate target; the target detector obtained after clustering processing is used as the target detector to be trained.

[0142] Optionally, a feature clustering algorithm such as KMeans may be used to cluster the features corresponding to the C categories of the last fully connected layer of the detection head.

[0143] Please refer to Figure 4. In one embodiment of the present invention, the detection head includes: a first fully connected layer (FC1), a Relu layer, a second fully connected layer (FC2), a Relu layer and a third fully connected layer (FC3) connected in sequence.

[0144] The above clustering process can be as follows: The input of the last fully connected layer FC3 of the detection head is N in =1024-dimensional vector, the output is N out = 4C-dimensional vector. The FC3 weight matrix W is 1024×4C, and the FC3 offset matrix B is 1024×C. Combining them yields a 1024×5C coefficient matrix, which is then reorganized to yield a 5120×C parameter matrix P. This matrix can be viewed as a 5120-dimensional feature corresponding to each category. Using a feature clustering algorithm (such as KMeans), C categories can be clustered into K categories, corresponding to K detection boxes. Categories in the same cluster share the same detection box.

[0145] Referring to FIG5 , an example of the present invention further provides a method for training an object detector, wherein the object detector includes a first feature extraction network, a candidate detection head, a second feature extraction network, and a detection head. The method includes:

[0146] Step 51: Acquire multiple data sets;

[0147] Optionally, each of the data sets includes multiple images and labels corresponding to the images, and the labels include categories and detection boxes of the marked objects.

[0148] Step 52: Copy the target detector to be trained to obtain a student detector and a teacher detector;

[0149] Step 53: inputting the target image in the target data set in the multiple data sets into the teacher detector to obtain a first detection target, and pseudo-labeling the first detection target that has not been manually labeled;

[0150] Step 54: input the target image into the first feature extraction network of the student detector to obtain a first feature image; input the first feature image into the candidate detection head of the student detector to obtain a target score and a candidate box of the candidate target; determine the target score loss of the candidate detection head according to the target score, and determine the candidate box loss of the candidate detection head according to the candidate box; input the candidate box and the first feature image into the second feature extraction network of the student detector to obtain a second feature image corresponding to the candidate box; input the second feature image into the detection head of the student detector to obtain a unified category score and multiple detection boxes of the detected target; select a target detection box corresponding to the category in the label space of the target data set from the multiple detection boxes; determine the detection head category score loss and pseudo-labeling loss according to the unified category score; determine the detection head detection box loss according to the target detection box;

[0151] Step 55: Optimize the target detector according to the candidate detection head target score loss, the candidate detection head candidate box loss, the detection head category score loss, the detection head detection box loss and the pseudo-labeling loss to obtain an optimized target detector.

[0152] In multi-dataset training, each dataset is only labeled with its corresponding label space L i The targets in the unified label space L are not labeled, and there are missing labeled targets. In order to maximize the use of the images in the dataset and provide as many labeled targets as possible for training, in an embodiment of the present invention, a teacher detector is used to detect the images in the dataset and automatically label the missing labels in the images. In order to distinguish them from the real labels manually labeled in the dataset, the labels of the targets labeled by the teacher detector are called pseudo-labels. The pseudo-labeling loss is calculated using the pseudo-labeling, and the total loss is obtained by adding it to the manual labeling loss obtained by manual labeling. This is used to train the student detector, further enriching the cross-scene labeled data, and thus improving the detection performance of the trained target detector.

[0153] In the embodiment of the present invention, optionally, the candidate detection head outputs a plurality of target scores for the candidate target, the number of which is the same as the number of the data sets, each target score corresponding to a data set, indicating the likelihood that the candidate target is a labeled target in the corresponding data set;

[0154] Determining a candidate detection head target score loss according to the target score includes:

[0155] Obtaining a target score of the candidate target corresponding to the target data set;

[0156] The target score loss of the candidate detection head is determined according to the target score of the candidate target corresponding to the target data set.

[0157] In an embodiment of the present invention, optionally, the unified category score includes multiple category scores, each category score corresponds to a category in a unified label space, and the unified label space records the categories of all labeled objects in the data set.

[0158] In the embodiment of the present invention, optionally, determining the detection head category score loss and the pseudo-labeling loss according to the unified category score includes:

[0159] Mapping the unified category score to a dataset category score, where the dataset category score is one or more, each of the dataset category scores corresponding to a category in the label space of the target dataset;

[0160] Determining a detection head category score loss based on the category score of the dataset;

[0161] A pseudo labeling loss is determined according to a pseudo category score in addition to the dataset category score in the unified category score.

[0162] Taking Figure 6 as an example, the unified category score C of the detection target output by the student detector is u , assuming that the target image is an image in the dataset D1, then the unified category score C u Mapped to dataset D1, we get the dataset category score C1 and the unified category score C u The category score C'1 in addition to the category score C1 of the dataset.

[0163] In an embodiment of the present invention, optionally, for each candidate target, the detection head outputs C detection boxes, where C is the number of categories in the unified label space, and each detection box corresponds to a category in the unified label space.

[0164] In an embodiment of the present invention, optionally, for each candidate target, the detection output has K detection boxes, and each of the detection boxes corresponds to one or more categories in the unified label space.

[0165] In the embodiment of the present invention, optionally, the object detector training method further includes:

[0166] The teacher detector is periodically updated using the student detector.

[0167] In the embodiment of the present invention, optionally, the periodically using the student detector to update the teacher detector includes:

[0168] When the learning rate of the target detector meets a preset condition, the student detector is used to update the teacher detector.

[0169] In an embodiment of the present invention, optionally, the preset condition is that the learning rate of the optimized target detector reaches the lowest point of the Cosine function.

[0170] In an embodiment of the present invention, the teacher detector and the student detector utilize a two-stage target detector with the same structure. During model training, the accuracy of the student detector gradually improves. Therefore, the student detector is periodically used to update the teacher detector, gradually improving the quality of pseudo-labels annotated by the teacher detector. Optionally, the student detector can be trained using a periodically varying Cosine learning rate during training. Whenever the learning rate reaches the minimum point of the Cosine function, the teacher detector is updated with the student detector. Practice has shown that this periodic updating of the teacher detector can achieve higher accuracy than real-time updating of the teacher detector.

[0171] The advantages of the object detector training method in the embodiment of the present invention are described below in conjunction with specific experimental results.

[0172] Experimental Results Comparison: See Table 1. Using the COCO-SPLIT5 dataset to train an object detector, the mAP (mean average accuracy) achieved without pseudo-labeling was 39.29, while using offline pseudo-labeling achieved a mAP of 41.81. Using the traditional semi-supervised learning EMA real-time update method achieved a mAP of 41.47. The semi-supervised learning periodic update method proposed in this paper achieved the highest mAP, reaching 42.57. Experimental results show that the training method of this paper significantly improves accuracy by over 1 percentage point compared to traditional semi-supervised learning methods.

[0173] Table 1

[0174] Referring to FIG7 , an embodiment of the present invention further provides a target detection method, which is applied to a target detector. The target detector includes a first feature extraction network, a candidate detection head, a second feature extraction network, and a detection head. The target detector is trained using the target detector training method described in any of the above embodiments. The target detection method includes:

[0175] Step 71: Input the image to be detected into the first feature extraction network to obtain a first feature image;

[0176] Step 72: Input the first feature image into the candidate detection head to obtain the target score and candidate box of the candidate target;

[0177] Step 73: Inputting the candidate box and the first feature image into a second feature extraction network to obtain a second feature image corresponding to the candidate box;

[0178] Step 74: Input the second feature image into the detection head to obtain a unified category score and a detection frame of the detection target, wherein the unified category score includes multiple category scores, each category score corresponds to a category in a unified label space, and the unified label space records the categories of the targets marked in multiple data sets.

[0179] In an embodiment of the present invention, optionally, the first feature image is input into a candidate detection head to obtain a target score and a candidate frame of the candidate target, including: the candidate detection head outputs multiple target scores for the candidate target, and the target score with the highest score is selected as the target score of the candidate target.

[0180] In an embodiment of the present invention, optionally, the detection head outputs C detection frames, where C is the number of categories in the unified label space, and each detection frame corresponds to a category in the unified label space.

[0181] In the embodiment of the present invention, optionally, the detection output has K detection boxes, and each of the detection boxes corresponds to one or more categories in the unified label space.

[0182] In the embodiment of the present invention, by designing a multi-detection frame detection head, accurate positioning of multiple categories of detection frames can be achieved.

[0183] The object detector training method and the object detection method according to the embodiments of the present invention can be applied to various fields.

[0184] For example, in one embodiment, it can be applied to the field of vehicle autonomous driving. In the field of autonomous driving, the target detector used for autonomous driving is required to be able to detect multiple types of targets, such as pedestrians, cars, buses, bicycles, electric vehicles, traffic lights, signs, puppies on the road, obstacles on the road, etc. When training the target detector, since there are many types of targets that need to be identified, it is difficult for one data set to cover all categories. Therefore, it is necessary to merge the label spaces of multiple data sets to obtain a unified label space, and use multiple data sets to train the target detector so that the trained target detector can recognize more categories.

[0185] In another embodiment, it can be applied to the field of video surveillance, such as the field of video surveillance for school cafeterias, which requires the target detector used for video surveillance to be able to detect multiple types of targets, such as people, pots and pans, mice, vegetables, seasonings, etc. When training the target detector, due to the wide variety of targets that need to be identified, it is difficult for one data set to cover all categories. Therefore, it is necessary to merge the label spaces of multiple data sets to obtain a unified label space, and use multiple data sets to train the target detector so that the trained target detector can recognize more categories. Please refer to Figure 8. An embodiment of the present invention also provides a training device 80 for a target detector, wherein the target detector includes a first feature extraction network, a candidate detection head, a second feature extraction network and a detection head. The training device for the target detector includes:

[0186] A first acquisition module 81 is used to acquire multiple data sets;

[0187] A first feature extraction module 82 is configured to input a target image in a target data set among the multiple data sets into a first feature extraction network to obtain a first feature image;

[0188] The candidate target detection module 83 is configured to input the first feature image into a candidate detection head to obtain a target score and a candidate box of the candidate target; determine a target score loss of the candidate detection head according to the target score, and determine a candidate box loss of the candidate detection head according to the candidate box;

[0189] A second feature extraction module 84 is configured to input the candidate frame and the first feature image into a second feature extraction network to obtain a second feature image corresponding to the candidate frame;

[0190] The target detection module 85 is configured to input the second feature image into a detection head to obtain a unified category score and multiple detection frames for the detected target; select a target detection frame corresponding to a category in the label space of the target dataset from the multiple detection frames; determine a detection head category score loss based on the unified category score; and determine a detection head detection frame loss based on the target detection frame.

[0191] The first optimization module 86 is used to optimize the object detector according to the candidate detection head target score loss, the candidate detection head candidate box loss, the detection head category score loss and the detection head detection box loss to obtain an optimized object detector.

[0192] Optionally, each of the data sets includes multiple images and labels corresponding to the images, and the labels include categories and detection boxes of the marked objects.

[0193] Optionally, the candidate detection head outputs a plurality of target scores for the candidate target, the number of which is the same as the number of the data sets, each target score corresponding to a data set, indicating the likelihood that the candidate target is a labeled target in the corresponding data set;

[0194] The candidate target detection module 83 is configured to obtain the target score of the candidate target corresponding to the target data set from the target score of the candidate target; and determine the target score loss of the candidate detection head according to the target score of the candidate target corresponding to the target data set.

[0195] Optionally, the unified category score includes multiple category scores, each category score corresponds to a category in a unified label space, and the unified label space records the categories of all labeled objects in the data set.

[0196] Optionally, the target detection module 85 is used to map the unified category score to a dataset category score, where the dataset category score is one or more, and each dataset category score corresponds to a category in the label space of the target dataset; and determine the detection head category score loss based on the dataset category score.

[0197] Optionally, for each candidate target, the detection head outputs C detection boxes, where C is the number of categories in the unified label space, and each detection box corresponds to a category in the unified label space.

[0198] Optionally, for each candidate target, the detection output has K detection boxes, and each of the detection boxes corresponds to one or more categories in the unified label space.

[0199] Optionally, the target detector training device 80 further includes:

[0200] A second acquisition module is configured to acquire an initial target detector, wherein the detection head of the initial target detector outputs C detection boxes for each candidate target, where C is the number of categories in the unified label space, and each detection box corresponds to a category in the unified label space;

[0201] A clustering module, configured to cluster features corresponding to C categories of the last fully connected layer of the detection head to obtain features of K categories;

[0202] Among them, the categories in the same cluster share a detection box;

[0203] The detection head of the target detector obtained after clustering processing outputs K detection boxes for each candidate target; the target detector obtained after clustering processing serves as the target detector to be trained.

[0204] Optionally, the detection head includes: a first fully connected layer, a Relu layer, a second fully connected layer, a Relu layer and a third fully connected layer connected in sequence.

[0205] Referring to FIG9 , an embodiment of the present invention further provides a training device 90 for an object detector. The object detector includes a first feature extraction network, a candidate detection head, a second feature extraction network, and a detection head. The training device for the object detector includes:

[0206] A first acquisition module 91 is used to acquire multiple data sets;

[0207] A copy module 92 is used to copy the target detector to be trained to obtain a student detector and a teacher detector;

[0208] a pseudo-labeling module 93, configured to input a target image in a target data set in the plurality of data sets into the teacher detector to obtain a first detection target, and pseudo-label the first detection target that has not been manually labeled;

[0209] The training module 94 is used to input the target image into the first feature extraction network of the student detector to obtain a first feature image; input the first feature image into the candidate detection head of the student detector to obtain a target score and a candidate frame of the candidate target; determine the target score loss of the candidate detection head according to the target score, and determine the candidate frame loss of the candidate detection head according to the candidate frame; input the candidate frame and the first feature image into the second feature extraction network of the student detector to obtain a second feature image corresponding to the candidate frame; input the second feature image into the detection head of the student detector to obtain a unified category score and multiple detection frames of the detection target; select a target detection frame corresponding to the category in the label space of the target data set from the multiple detection frames; determine the detection head category score loss and pseudo-labeling loss according to the unified category score; determine the detection head detection frame loss according to the target detection frame; optimize the target detector according to the candidate detection head target score loss, the candidate detection head candidate frame loss, the detection head category score loss, the detection head detection frame loss and the pseudo-labeling loss to obtain an optimized target detector.

[0210] Optionally, each of the data sets includes multiple images and labels corresponding to the images, and the labels include categories and detection boxes of the marked objects.

[0211] Optionally, the candidate detection head outputs a plurality of target scores for the candidate target, the number of which is the same as the number of the data sets, each target score corresponding to a data set, indicating the likelihood that the candidate target is a labeled target in the corresponding data set;

[0212] The training module 94 is configured to obtain the target score of the candidate target corresponding to the target data set; and determine the target score loss of the candidate detection head according to the target score of the candidate target corresponding to the target data set.

[0213] Optionally, the unified category score includes multiple category scores, each category score corresponds to a category in a unified label space, and the unified label space records the categories of all labeled objects in the data set.

[0214] Optionally, the training module 94 is used to map the unified category score to a dataset category score, where the dataset category score is one or more, and each dataset category score corresponds to a category in the label space of the target dataset; determine the detection head category score loss based on the dataset category score; and determine the pseudo-labeling loss based on the pseudo-category score in the unified category score other than the dataset category score.

[0215] Optionally, for each candidate target, the detection head outputs C detection boxes, where C is the number of categories in the unified label space, and each detection box corresponds to a category in the unified label space.

[0216] Optionally, for each candidate target, the detection output has K detection boxes, and each of the detection boxes corresponds to one or more categories in the unified label space.

[0217] Optionally, the target detector training device 90 further includes:

[0218] An updating module is used to periodically update the teacher detector using the student detector.

[0219] Optionally, the updating module is configured to update the teacher detector using the student detector when the learning rate of the target detector meets a preset condition.

[0220] Optionally, the preset condition is that the learning rate of the optimized target detector reaches the lowest point of the Cosine function.

[0221] Referring to FIG. 10 , an embodiment of the present invention further provides an object detection device 100 that detects objects using an object detector. The object detector includes a first feature extraction network, a candidate detection head, a second feature extraction network, and a detection head. The object detection device 100 includes:

[0222] A first feature extraction module 101 is configured to input an image to be detected into a first feature extraction network of an object detector to obtain a first feature image;

[0223] The candidate target detection module 102 is configured to input the first feature image into a candidate detection head to obtain a target score and a candidate frame of the candidate target;

[0224] A second feature extraction module 103 is configured to input the candidate frame and the first feature image into a second feature extraction network to obtain a second feature image corresponding to the candidate frame;

[0225] The target detection module 104 is used to input the second feature image into the detection head to obtain a unified category score and a detection frame of the detected target, wherein the unified category score includes multiple category scores, each category score corresponds to a category in a unified label space, and the unified label space records the categories of targets marked in multiple data sets.

[0226] Optionally, the candidate detection head outputs multiple target scores for the candidate target, and the candidate target detection module 102 is configured to select the target score with the highest score as the target score for the candidate target.

[0227] Optionally, the detection head outputs C detection frames, where C is the number of categories in the unified label space, and each detection frame corresponds to a category in the unified label space.

[0228] Optionally, the detection output has K detection boxes, and each detection box corresponds to one or more categories in the unified label space.

[0229] Please refer to Figure 11. An embodiment of the present invention also provides an electronic device 110, including a processor 111, a memory 112, and a computer program stored in the memory 112 and executable on the processor 111. When the computer program is executed by the processor 111, it implements the above-mentioned target detector training method or each process of the target detection method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described here.

[0230] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the computer program implements the above-mentioned target detector training method or each process of the target detection method embodiment, and can achieve the same technical effect. To avoid repetition, it is not described here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0231] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0232] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0233] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are protected by the present invention.

Claims

1. A method for training an object detector, wherein: The object detector includes a first feature extraction network, a candidate detection head, a second feature extraction network and a detection head, and the method includes: Get multiple datasets; Inputting a target image in a target data set among the multiple data sets into a first feature extraction network to obtain a first feature image; Inputting the first feature image into a candidate detection head to obtain a target score and a candidate frame of the candidate target; determining a target score loss of the candidate detection head according to the target score, and determining a candidate frame loss of the candidate detection head according to the candidate frame; Inputting the candidate box and the first feature image into a second feature extraction network to obtain a second feature image corresponding to the candidate box; Inputting the second feature image into the detection head to obtain a unified category score and multiple detection frames of the detected target; selecting a target detection frame corresponding to the category in the label space of the target data set from the multiple detection frames; determining the detection head category score loss according to the unified category score; determining the detection head detection frame loss according to the target detection frame; The target detector is optimized according to the candidate detection head target score loss, the candidate detection head candidate frame loss, the detection head category score loss and the detection head detection frame loss to obtain an optimized target detector.

2. The method according to claim 1, wherein: Each of the data sets includes multiple images and labels corresponding to the images, and the labels include categories and detection boxes of annotated objects.

3. The method according to claim 1, wherein: The candidate detection head outputs a plurality of target scores for the candidate targets, the number of which is the same as the number of the data sets, each target score corresponding to a data set, indicating the likelihood that the candidate target is a labeled target in the corresponding data set; Determining a candidate detection head target score loss according to the target score includes: Obtaining a target score of the candidate target corresponding to the target data set; The target score loss of the candidate detection head is determined according to the target score of the candidate target corresponding to the target data set.

4. The method according to claim 1, wherein: The unified category score includes multiple category scores, each category score corresponds to a category in a unified label space, and the unified label space records the categories of all labeled objects in the data set.

5. The method according to claim 4, wherein: The step of determining the detection head category score loss according to the unified category score comprises: Mapping the unified category score to a dataset category score, where the dataset category score is one or more, and each of the dataset category scores corresponds to a category in the label space of the target dataset; The detection head category score loss is determined according to the dataset category score.

6. The method according to claim 4, wherein: For each candidate target, the detection head outputs C detection frames, where C is the number of categories in the unified label space, and each detection frame corresponds to a category in the unified label space.

7. The method according to claim 4, wherein: For each candidate target, the detection output has K detection boxes, and each of the detection boxes corresponds to one or more categories in the unified label space.

8. The method according to claim 7, wherein: Also includes: Obtain an initial target detector, wherein a detection head of the initial target detector outputs C detection frames for each candidate target, where C is the number of categories in the unified label space, and each detection frame corresponds to a category in the unified label space; Clustering the features corresponding to the C categories of the last fully connected layer of the detection head to obtain features of K categories; Among them, categories in the same cluster share a detection box; The detection head of the target detector obtained after the clustering process outputs K detection frames for each candidate target; the target detector obtained after the clustering process is used as the target detector to be trained.

9. The method according to claim 1 or 8, wherein: The detection head includes: a first fully connected layer, a Relu layer, a second fully connected layer, a Relu layer and a third fully connected layer connected in sequence.

10. A method for training a target detector, wherein: The object detector includes a first feature extraction network, a candidate detection head, a second feature extraction network and a detection head, and the method includes: Get multiple datasets; The target detector to be trained is copied to obtain a student detector and a teacher detector; Inputting a target image in a target data set in the multiple data sets into the teacher detector to obtain a first detection target, and pseudo-labeling the first detection target that has not been manually labeled; Input the target image into the first feature extraction network of the student detector to obtain a first feature image; input the first feature image into the candidate detection head of the student detector to obtain a target score and a candidate frame of the candidate target; determine the target score loss of the candidate detection head according to the target score, and determine the candidate frame loss of the candidate detection head according to the candidate frame; input the candidate frame and the first feature image into the second feature extraction network of the student detector to obtain a second feature image corresponding to the candidate frame; input the second feature image into the detection head of the student detector to obtain a unified category score and multiple detection frames of the detected target; select a target detection frame corresponding to the category in the label space of the target data set from the multiple detection frames; determine the detection head category score loss and pseudo-labeling loss according to the unified category score; determine the detection head detection frame loss according to the target detection frame; The target detector is optimized according to the candidate detection head target score loss, the candidate detection head candidate box loss, the detection head category score loss, the detection head detection box loss and the pseudo-labeling loss to obtain an optimized target detector.

11. The method according to claim 10, wherein: Each of the data sets includes multiple images and labels corresponding to the images, and the labels include categories and detection boxes of annotated objects.

12. The method according to claim 10, wherein: The candidate detection head outputs a plurality of target scores for the candidate targets, the number of which is the same as the number of the data sets, each target score corresponding to a data set, indicating the likelihood that the candidate target is a labeled target in the corresponding data set; Determining a candidate detection head target score loss according to the target score includes: Obtaining a target score of the candidate target corresponding to the target data set; The target score loss of the candidate detection head is determined according to the target score of the candidate target corresponding to the target data set.

13. The method according to claim 10, wherein: The unified category score includes multiple category scores, each category score corresponds to a category in a unified label space, and the unified label space records the categories of all labeled objects in the data set.

14. The method according to claim 13, wherein: The determining of the detection head category score loss and the pseudo-labeling loss according to the unified category score includes: Mapping the unified category score to a dataset category score, where the dataset category score is one or more, and each of the dataset category scores corresponds to a category in the label space of the target dataset; Determining a detection head category score loss according to the category score of the data set; A pseudo-labeling loss is determined according to a pseudo-category score in the unified category score except for the dataset category score.

15. The method according to claim 13, wherein: For each candidate target, the detection head outputs C detection frames, where C is the number of categories in the unified label space, and each detection frame corresponds to a category in the unified label space.

16. The method according to claim 15, wherein: For each candidate target, the detection output has K detection boxes, and each of the detection boxes corresponds to one or more categories in the unified label space.

17. The method according to claim 10, wherein: Also includes: The teacher detector is periodically updated using the student detector.

18. The method according to claim 17, wherein: The periodic use of the student detector to update the teacher detector comprises: When the learning rate of the target detector meets a preset condition, the student detector is used to update the teacher detector.

19. The method according to claim 18, wherein: The preset condition is that the learning rate of the optimized target detector reaches the lowest point of the Cosine function.

20. A target detection method, applied to a target detector, the target detector comprising a first feature extraction network, a candidate detection head, a second feature extraction network and a detection head, the method comprising: Inputting the image to be detected into a first feature extraction network to obtain a first feature image; Inputting the first feature image into a candidate detection head to obtain a target score and a candidate box of the candidate target; Inputting the candidate box and the first feature image into a second feature extraction network to obtain a second feature image corresponding to the candidate box; The second feature image is input into the detection head to obtain a unified category score and a detection frame of the detected target, wherein the unified category score includes multiple category scores, each category score corresponds to a category in a unified label space, and the unified label space records the categories of targets marked in multiple data sets.

21. The method according to claim 20, wherein: The step of inputting the first feature image into a candidate detection head to obtain a target score and a candidate frame of the candidate target includes: The candidate detection head outputs multiple target scores for the candidate target, and the target score with the highest score is selected as the target score for the candidate target.

22. A training device for a target detector, wherein: The target detector includes a first feature extraction network, a candidate detection head, a second feature extraction network and a detection head, and the training device of the target detector includes: A first acquisition module, used to acquire multiple data sets; A first feature extraction module, used for inputting a target image in a target data set in the multiple data sets into a first feature extraction network to obtain a first feature image; A candidate target detection module is used to input the first feature image into a candidate detection head to obtain a target score and a candidate frame of the candidate target; determine a target score loss of the candidate detection head according to the target score, and determine a candidate frame loss of the candidate detection head according to the candidate frame; A second feature extraction module, used for inputting the candidate box and the first feature image into a second feature extraction network to obtain a second feature image corresponding to the candidate box; The target detection module is used to input the second feature image into the detection head to obtain a unified category score and multiple detection frames of the detected target; select a target detection frame corresponding to the category in the label space of the target data set from the multiple detection frames; determine the detection head category score loss according to the unified category score; determine the detection head detection frame loss according to the target detection frame; The first optimization module is used to optimize the target detector according to the candidate detection head target score loss, the candidate detection head candidate frame loss, the detection head category score loss and the detection head detection frame loss to obtain an optimized target detector.

23. A training device for a target detector, wherein: The target detector includes a first feature extraction network, a candidate detection head, a second feature extraction network and a detection head, and the training device of the target detector includes: A first acquisition module, used to acquire multiple data sets; A copy module is used to copy the target detector to be trained to obtain a student detector and a teacher detector; A pseudo-annotation module, used for inputting a target image in a target data set in the multiple data sets into the teacher detector to obtain a first detection target, and pseudo-annotating the first detection target that has not been manually annotated; A training module is used to input the target image into the first feature extraction network of the student detector to obtain a first feature image; input the first feature image into the candidate detection head of the student detector to obtain a target score and a candidate frame of the candidate target; determine the target score loss of the candidate detection head according to the target score, and determine the candidate frame loss of the candidate detection head according to the candidate frame; input the candidate frame and the first feature image into the second feature extraction network of the student detector to obtain a second feature image corresponding to the candidate frame; input the second feature image into the detection head of the student detector to obtain a unified category score and multiple detection frames of the detected target; select a target detection frame corresponding to a category in the label space of the target data set from the multiple detection frames; determine the detection head category score loss and pseudo-labeling loss according to the unified category score; determine the detection head detection frame loss according to the target detection frame; optimize the target detector according to the candidate detection head target score loss, the candidate detection head candidate frame loss, the detection head category score loss, the detection head detection frame loss and the pseudo-labeling loss to obtain an optimized target detector.

24. A target detection device, using a target detector to detect a target, the target detector comprising a first feature extraction network, a candidate detection head, a second feature extraction network and a detection head, the target detection device comprising: A first feature extraction module, used for inputting the image to be detected into a first feature extraction network of the target detector to obtain a first feature image; A candidate target detection module, used to input the first feature image into a candidate detection head to obtain a target score and a candidate frame of the candidate target; A second feature extraction module, used for inputting the candidate box and the first feature image into a second feature extraction network to obtain a second feature image corresponding to the candidate box; The target detection module is used to input the second feature image into the detection head to obtain a unified category score and a detection frame of the detected target, wherein the unified category score includes multiple category scores, each of which is Each category score corresponds to a category in a unified label space, which records the categories of objects annotated in multiple datasets.

25. An electronic device comprising: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the training method for a target detector as described in any one of claims 1 to 9, or, when executed by the processor, implements the steps of the training method for a target detector as described in any one of claims 10 to 19, or, when executed by the processor, implements the steps of the target detection method as described in claim 20 or 21.

26. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the target detector training method as described in any one of claims 1 to 9; or, when executed by a processor, implements the steps of the target detector training method as described in any one of claims 10 to 19; or, when executed by a processor, implements the steps of the target detection method as described in claim 20 or 21.

Images