Target detection method and device, electronic equipment and storage medium

Abstract

The invention provides a target detection method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a first data set and a second data set, the first data set comprises a plurality of sample images with labels, and the second data set comprises a plurality of sample images without labels; a first target detection model is obtained, the first target detection model comprises at least two detection heads, different detection heads correspond to different detection target groups, and the detection target groups comprise at least two different types of detection targets; and the detection heads are used for detecting the detection targets included in the corresponding detection target groups. And training the first target detection model through the first data set to obtain a second target detection model. And training the second target detection model according to the first data set and the second data set to obtain a third target detection model. According to the scheme, the accuracy of target detection can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing, and in particular to a target detection method, device, electronic device and storage medium. Background Art

[0002] With the widespread use of deep learning in artificial intelligence computer vision, more and more researchers are applying deep learning to the field of autonomous driving. Data collection and labeling take up about 80% of the time in artificial intelligence and machine learning projects. The existing data labeling method is mainly pre-labeling, that is, using a trained deep learning network to detect and label objects in the image.

[0003] At present, in order to improve the accuracy of target detection, Dropout technology is used in machine learning to randomly inactivate hidden neurons in the network in a certain proportion, so that the target detection model will not be too dependent on certain local features.

[0004] However, when the amount of data is small and the quality is not high, the above methods have limited ability to improve the generalization of the model, and the target detection model still suffers from serious overfitting, resulting in inaccurate detection of the target detection model. Summary of the invention

[0005] In view of this, embodiments of the present application provide a target detection method, device, electronic device and storage medium to at least solve some of the above problems.

[0006] According to a first aspect of an embodiment of the present application, a target detection method is provided, comprising: obtaining a first data set and a second data set, wherein the first data set includes a plurality of labeled sample images, and the second data set includes a plurality of unlabeled sample images. Obtaining a first target detection model, wherein the first target detection model includes at least two detection heads, different detection heads correspond to different detection target groups, the detection target groups include at least two different categories of detection targets, and the detection heads are used to detect the detection targets included in the corresponding detection target groups. The first target detection model is trained using the first data set to obtain a second target detection model. The second target detection model is trained based on the first data set and the second data set to obtain a third target detection model. The third target detection model is used to perform target detection on the image to be detected.

[0007] According to a second aspect of an embodiment of the present application, a target detection device is provided, comprising: a data acquisition module, used to acquire a first data set and a second data set, wherein the first data set includes a plurality of labeled sample images, and the second data set includes a plurality of unlabeled sample images. An acquisition module, used to acquire a first target detection model, wherein the first target detection model includes at least two detection heads, different detection heads correspond to different detection target groups, the detection target groups include at least two different categories of detection targets, and the detection heads are used to detect the detection targets included in the corresponding detection target groups. A first training module, used to train the first target detection model using the first data set to obtain a second target detection model. A second training module, used to train the second target detection model based on the first data set and the second data set to obtain a third target detection model. A target detection module, used to perform target detection on the image to be detected using the third target detection model.

[0008] According to a third aspect of the embodiments of the present application, an electronic device is provided, comprising: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the target detection method described in the first aspect above.

[0009] According to a fourth aspect of an embodiment of the present application, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the target detection method as described in the first aspect above is implemented.

[0010] According to the above technical solution, a first data set including labeled sample images and a second data set including unlabeled sample images are obtained, the first target detection model includes at least two detection heads, different detection heads correspond to different detection target groups, the detection target groups include at least two different categories of detection targets, the detection heads are used to detect the detection targets included in the corresponding detection target groups, the first target detection model is trained by the first data set to obtain the second target detection model, and then the second target detection model is trained according to the first data set and the second data set to obtain the third target detection model, and the target detection is performed on the image to be detected by the third target detection model. The first target detection model includes at least two detection heads, and each detection head corresponds to a different detection target group. Each detection head only needs to detect the detection targets included in the corresponding detection target group. Different detection targets do not interfere with each other, which improves the accuracy of the model in target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0012] Figure 1 is a flow chart of target detection provided by an embodiment of the present application;

[0013] Figure 2 is a schematic diagram of target detection provided by another embodiment of the present application;

[0014] Figure 3 is a schematic diagram of a target detection device provided by an embodiment of the present application;

[0015] Figure 4 It is a schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0016] The present application is described below based on embodiments, but the present application is not limited to these embodiments. In the detailed description of the present application below, some specific details are described in detail. It is possible for a person skilled in the art to fully understand the present application without the description of these details. In order to avoid confusing the essence of the present application, known methods, processes, and flows are not described in detail. In addition, the drawings are not necessarily drawn to scale.

[0017] Object Detection Methods

[0018] The present application provides a target detection method, which is described in detail below through multiple embodiments.

[0019] Figure 1 FIG. 1 is a flow chart of a target detection method according to an embodiment of the present application. Figure 1 As shown, the target detection method includes the following steps:

[0020] Step S102: Acquire a first data set and a second data set.

[0021] The multiple sample images included in the first data set and the second data set may be collected by means of a camera device, an infrared sensor, etc., and this embodiment of the present application does not impose any limitation on this.

[0022] The first data set includes multiple labeled sample images. The collected sample images are manually annotated according to the annotation specifications, and a label is assigned to each sample image. The label contains the category information and location coordinates represented by the sample image. After manual annotation, the first data set is divided into a training set, a validation set, and a test set according to a certain ratio.

[0023] The second dataset includes multiple unlabeled sample images, which are not annotated for the collected sample images. They usually contain original images but lack clear labels to identify the categories and location coordinates of the sample images.

[0024] In order to balance the data, the ratio of the first data set to the second data set can be controlled between 1:3 and 1:10.

[0025] Step S104: Acquire a first target detection model.

[0026] Before training, the model structure is improved so that the first target detection model includes at least two detection heads, different detection heads correspond to different detection target groups, the detection target groups include at least two different categories of detection targets, and the detection heads are used to detect the detection targets included in the corresponding detection target groups. For example, the first target detection model includes two detection heads, the detection targets included in the detection target group corresponding to the first detection head are cats and dogs, and the detection targets included in the detection target group corresponding to the second detection head are cars and trucks.

[0027] The first target detection model can be obtained by training based on the YOLOV7 deep learning model. First, a public target detection data set, such as COCO and VOC data sets, is obtained. Then, an improved YOLOV7 is trained based on the public data set to obtain the first target detection model. It should be noted that YOLOV7 is only exemplary, and other deep learning models with target detection capabilities can also be selected, which is not limited in the embodiments of the present application.

[0028] Step S106: training the first target detection model using the first data set to obtain a second target detection model.

[0029] The first data set includes multiple labeled sample images. Training the first target detection model using the first data set is supervised training. The first target detection model learns the relationship between input and output through known input sample images and corresponding output labels, thereby obtaining a second target detection model that can perform target detection on sample images.

[0030] Step S108: training the second object detection model according to the first data set and the second data set to obtain a third object detection model.

[0031] The first data set includes labeled sample images, and the second data set includes unlabeled sample images. The first data set and the second data set are jointly used to train the model, namely semi-supervised training. In the case of a limited labeled first data set, the second data set including unlabeled sample images can be used to enhance the generalization ability of the model.

[0032] Step S110: performing object detection on the image to be detected by using a third object detection model.

[0033] The third target detection model obtained after supervised training and semi-supervised training has target detection capability and strong generalization ability, and can be used for target detection on images to be detected.

[0034] In an embodiment of the present application, a first data set including labeled sample images and a second data set of unlabeled sample images are obtained, a first target detection model includes at least two detection heads, different detection heads correspond to different detection target groups, the detection target groups include at least two different categories of detection targets, the detection heads are used to detect the detection targets included in the corresponding detection target groups, the first target detection model is trained with the first data set to obtain the second target detection model, and then the second target detection model is trained based on the first data set and the second data set to obtain the third target detection model, and the target detection is performed on the image to be detected with the third target detection model. The first target detection model includes at least two detection heads, and each detection head corresponds to a different detection target group, each detection head only needs to detect the detection targets included in the corresponding detection target group, and different detection targets do not interfere with each other, thereby improving the accuracy of the model in performing target detection.

[0035] In one possible implementation, based on the detection target group corresponding to the detection head and the label of the sample image in the first data set, feature extraction is performed on the sample image with different downsampling multiples to obtain at least two labeled feature maps, and the feature map with the corresponding label in the target detection group corresponding to the detection head is determined as the training data corresponding to the detection head, wherein different feature maps correspond to different resolutions. Data, the detection head is trained using the training data corresponding to the detection head.

[0036] When training the first model, not only the detection head included in the training model, but also the backbone network and Neck network of the model need to be trained. When training the model, in the labeled first data set, a sample image may include multiple detection targets. All sample images are reset to a fixed size in a certain way as model input. The backbone network and Neck network of the model extract features from the input features to obtain feature maps of different resolutions. These feature maps are then sent to the corresponding detection heads as training data. The detection heads will calculate the loss based on the manually labeled true value, that is, if the true value is included in the detection target range of the detection head, the loss is calculated in this detection head, and then back propagation is performed to continuously update the network parameters of the backbone network, Neck network and detection head.

[0037] In an embodiment of the present application, the detection target group corresponding to the detection head includes different detection targets, and the training data of each detection head only includes the detection targets included in the corresponding detection target group. If the category corresponding to the label of the training data is not within the detection targets included in the detection target group corresponding to the detection head, the detection head does not learn the features of the training data. By accurately training the model, the accuracy of the model in target detection is improved.

[0038] In one possible implementation, the detection head includes at least two sub-detection heads, different sub-detection heads correspond to different resolutions, and the step of training the detection head through corresponding training data includes: according to the resolution corresponding to the sub-detection head, inputting the feature map with the same resolution corresponding to the sub-detection head into the sub-detection head for target detection, so as to train the model.

[0039] Each detection head includes at least two sub-detection heads. When training the target detection model, different sub-detection heads are responsible for detecting feature maps with different resolutions. When training the model, the resolution of the sample images may not be the same. For example, some images are 1920×1080, while others are 1280×1280. When the model needs to perform matrix calculations, the size of the input image should remain consistent. Therefore, the conventional practice is to reset all sample images to a fixed size (such as 1280×1280) according to a certain method (such as linear interpolation). Then data enhancement can be performed, and the image features obtained are passed to the backbone network of the model (such as ResNet) for feature extraction. The extracted features are obtained according to the different downsampling multiples (generally 8 / 16 / 32 times downsampling) to obtain feature maps of different resolutions. Feature maps of different resolutions are sent to different sub-detection heads for detection. For example, the target detection model has two detection heads, detection head a is responsible for detecting cars, and detection head b is responsible for detecting bicycles and tricycles. Each detection head includes three sub-detection heads. The three sub-detection heads corresponding to different detection heads are responsible for detecting feature maps corresponding to large resolution 160×160, medium resolution 80×80 and small resolution 40×40 respectively. After the original sample image is reset to 1280×1280×3 (3 is the number of RGB channels), it is downsampled by 8 times to obtain a feature map of 160×160. For example, the 1280×1280×3 image is downsampled by 8 / 16 / 32 times, respectively, to obtain The feature maps of the car in three dimensions of 160×160×C, 80×80×C, and 40×40×C are converted into feature maps, where C is the number of channels, which is determined by the number of convolution kernels. When training the model, the feature map of the car of 160×160×C will enter the sub-detection head with a resolution of 160×160 in detection head a, the feature map of the car of 80×80×C will enter the sub-detection head with a resolution of 80×80 in detection head a, and the feature map of the car of 40×40×C will enter the sub-detection head with a resolution of 40×40 in detection head a. The corresponding sub-detection heads are trained through feature maps of different resolutions.

[0040] In an embodiment of the present application, the detection head includes at least two sub-detection heads, and different sub-detection heads correspond to different resolutions. When training the model, the detection targets included in the training data are located in the target detection group corresponding to the detection head into which the training data enters. Furthermore, the sub-detection heads of the detection head are only responsible for learning the training data with the same resolution as that corresponding to the sub-detection head. By learning the training data with different resolutions through different sub-detection heads, the efficiency of model training is improved.

[0041] After the model training is completed, target detection is performed on the image to be detected through the third target detection model. The image to be detected is reset to a fixed resolution (such as 1280×1280) in a certain way (such as linear interpolation). Then, the backbone network and Neck network of the model extract features from the input image to obtain feature maps of different resolutions corresponding to the sub-detection heads included in different detection heads. After that, the sub-detection head calculates the position and category of the target box of the object that the sub-detection head needs to detect based on the input feature map and the network parameters saved by the third target detection model. For example, the image to be detected can be reset to a fixed size of 1280×1280 through linear interpolation, and then the backbone network and Neck network of the model extract features from the input image. The extracted features are divided into feature maps of different resolutions according to the different downsampling multiples (generally 8 / 16 / 32 times downsampling). For example, a 1280×1280×3 image is downsampled by 8 / 16 / 32 times to obtain feature maps of 160×160×C, 80×80×C and 40×40×C dimensions, respectively. Then the detection head The included sub-detection head calculates the position of the target frame and the detection target of the object to be detected by the sub-detection head according to the input feature map and the network parameters saved by the third target detection model. Different sub-detection heads correspond to different resolutions. For the feature map with a resolution of 80×80×C extracted, the sub-detection head with a resolution of 80×80 and responsible for detecting cats only regresses whether there is a cat. Other sub-detection heads with different resolutions and responsible for different detection targets also only regress whether there is a corresponding detection target for the feature map with the same resolution and the same detection target as the sub-detection head. In a possible implementation, Figure 2 As shown, the second target detection model is trained according to the first data set and the second data set to obtain the third target detection model, including:

[0042] Step S202: Determine the second target detection model as the model to be trained.

[0043] The second target detection model is trained based on the first data set and already has target detection capability. The second target detection model is determined as the model to be trained for further training.

[0044] Step S204: performing target detection on the sample images in the second data set using the model to be trained to obtain pseudo labels for the sample images in the second data set.

[0045] The sample images in the second data set are subjected to target detection by the model to be trained, and pseudo labels of all detected targets included in the sample images in the second data set are obtained, wherein the pseudo labels include probability values, categories and position coordinates corresponding to the detected targets. If the maximum probability value among the probability values ​​corresponding to all detected targets included in the sample image is greater than a preset threshold, the pseudo label of the detected target corresponding to the maximum probability value is determined as the pseudo label of the sample image.

[0046] Step S206: Generate an intermediate data set including sample images in the second data set and corresponding pseudo labels.

[0047] The sample images in the second data set are unlabeled, and the sample images in the second data set and the corresponding pseudo labels obtained in step 204 are reconstructed into an intermediate data set including the pseudo-labeled sample images.

[0048] Step S208: Train the second target detection model using the first data set and the intermediate data set to obtain an intermediate model.

[0049] Before training the second target detection model, sample images in the intermediate data set may be randomly extracted in proportion to perform data enhancement. The data enhancement method may include adding motion blur and performing weather transformation. The weather transformation at least includes extreme weather.

[0050] In some application scenarios, such as when collecting vehicle images, the captured images may be elongated or jittery due to the high speed of the vehicle. Randomly adding motion blur to the collected sample images at a certain ratio can increase the number of sample images and improve the generalization of the model.

[0051] There are also some application scenarios. For example, in extreme weather conditions, data collection is difficult and costly, so the data in this area is relatively small. We can randomly select 1 to 3 extreme weather conditions to enhance the collected sample images, such as adding cloudy, foggy, rainy, snowy, and snowy weather to the original photos. The intensity of the weather can be randomly selected. For example, when adding a rainy day, the size of the rain can be added through a random function. In addition, in extreme weather conditions, water or mud may splash from the windows or bodies of cars. We can also add such enhancement methods to sample images to simulate the scenario, improve the accuracy of target detection in extreme weather conditions, and further improve the generalization of the model.

[0052] The intermediate data set is assigned a certain weight, and the second target detection model is trained jointly with the first data set to obtain the intermediate model.

[0053] Step S210: Determine whether the performance of the intermediate model no longer increases, if so, execute step S212, otherwise execute step S214.

[0054] Before training, the intermediate model will set the number of training rounds, that is, the number of training rounds. The intermediate model will terminate only after completing these training rounds. Then the problem may be that the number of training rounds is set too small, and the performance of the intermediate model is still rising, which means that MAP0.5 and MAP0.5:0.9 are still rising. That is, during the training process, the intermediate model will continuously update the maximum weighted sum of MAP0.5 and MAP0.5:0.9 and the optimal parameters of the model. After each round of training, it will be verified on the validation set according to the parameters of the current model, and the weighted sum of MAP0.5 and MAP0.5:0.9 will be calculated. If the weighted sum of MAP0.5 and MAP0.5:0.9 of this round is greater than the weighted sum of the maximum MAP0.5 and MAP0.5:0.9 saved before, the maximum weighted sum of MAP0.5 and MAP0.5:0.9 will be replaced, and the optimal parameters of the model will also be replaced. The training can be terminated when the performance of the intermediate model is still rising, so a larger number of training rounds can be set in advance. During the training process, if the weighted sum of MAP0.5 and MAP0.5:0.9 no longer increases after a round of training, and it is observed that the loss function value no longer decreases, then the intermediate model training has reached the optimum. The model training can be manually terminated and step S212 is executed. Otherwise, step S214 is executed to continue model training.

[0055] Step S212: determine the intermediate model as the third target detection model and end the current process.

[0056] If the performance of the intermediate model no longer improves, it indicates that the model training has reached the optimal level, that is, the accuracy of the model in target detection for sample images in the validation set no longer improves. At this time, the training is terminated and the intermediate model is determined as the third target detection model.

[0057] Step S214: determine the intermediate model as the model to be trained.

[0058] If the performance of the intermediate model is increasing, it indicates that the model is insufficiently trained and underfitting occurs. The intermediate model needs to be determined as the model to be trained, and the training process of step S204 to step S210 needs to be performed until the third target detection model is obtained.

[0059] In an embodiment of the present application, an intermediate data set including sample images and corresponding pseudo labels in the second data set and the first data set are generated to train the model to be trained multiple times, and the training is terminated until the model performance is optimal, and finally a third target detection model is obtained. Through multiple trainings, the model learns as many features of the detection target as possible, thereby improving the accuracy of the model in target detection.

[0060] In a possible implementation, the category weight of the corresponding detection target is determined according to the number of detection targets included in the sample image in the first data set, wherein the category weight of the detection target is negatively correlated with the number of the detection target included in the first data set. The sample weight of the sample image is calculated according to the category weight of the detection target included in the sample image in the first data set, and the sample image in the first data set is loaded according to the sample weight of the sample image, and the first target detection model is trained, wherein the number of times the sample image is loaded is positively correlated with the sample weight of the sample image.

[0061] Since the collected data has serious data imbalance, the model is mainly dominated by the main detection targets with a large amount of data, while ignoring the detection targets at the tail, that is, the "long tail effect", resulting in low target detection accuracy. Therefore, the data imbalance phenomenon can be improved from the data direction.

[0062] The category weight of the detection target is negatively correlated with the number of the detection target included in the first data set, that is, the smaller the number of the detection target included in the first data set, the larger the corresponding category weight. For example, if the detection targets only include two categories, cars and people, and the training set has a total of 10 images, including 30 images of cars and 50 images of people, then the category weight of cars is calculated as: (1 / 30) / (1 / 30+1 / 50)*2, where 2 represents the number of categories of the detection target.

[0063] The sample weight of the sample image is calculated according to the category weights of the detection targets included in the sample image in the first data set, and the sample weight of the sample image is equal to the sum of the category weights of the images of all detection targets included in the sample image. For example, if there are 2 cars and 3 people in a sample image, the category weight of the car is 0.3, and the category weight of the person is 0.5, then the sample weight of this sample image is calculated as: 2*0.3+3*0.5.

[0064] The number of times a sample image is loaded is positively correlated with the sample weight of the sample image, that is, the larger the sample weight of the sample image, the more times the sample image is loaded during model training, so that the model can learn more features of the sample image.

[0065] In an embodiment of the present application, the category weight of the corresponding detection target is determined based on the number of detection targets included in the sample image in the first data set, so that the fewer the number of detection targets included in the first data set, the larger the category weight of the corresponding detection target. The sample weight of the sample image is further calculated based on the category weight. The larger the sample weight of the sample image, the more times the model loads the sample image, so that different categories of data can be balanced when training the model, thereby improving the accuracy of the model in target detection.

[0066] In one possible implementation, the loss function of the first target detection model includes:

[0067] FL α =-α(1- t ) γ log( t )

[0068] FL α A loss function for characterizing a first target detection model for a sample image including a detection target in a current detection target group; It is used to characterize the result of target detection performed by the first target detection model on the sample image including the detection targets in the current detection target group. The result of target detection is used to indicate the probability that the sample image includes each detection target in the current detection target group. y is used to characterize the label corresponding to the detection target included in the sample image in the first data set. α is used to characterize the loss weight corresponding to the current detection target group. γ is used to characterize the attention corresponding to the current detection target group. γ is positively correlated with the detection difficulty value of the detection target included in the current detection target group.

[0069] γ is positively correlated with the detection difficulty values ​​of the detection targets included in the current detection target group. The greater the detection difficulty value of the detection target, the greater γ.

[0070] During the training process, the imbalance of sample numbers can also be reduced by setting the loss weight of each detection target. When calculating the loss of each detection target, the loss weight of the detection target with a small number of sample images is increased, thereby increasing the model's penalty on these sample images and alleviating the imbalance problem. Focal loss can focus on difficult-to-predict sample images that are prone to detection errors, and balanced cross entropy loss function can focus on sample imbalance problems. The combination of the two generates the α variant loss function FL α The above two problems are adjusted simultaneously to achieve a more accurate target detection effect.

[0071] In the embodiment of the present application, the α variant loss function Fl α At the same time, it solves the problem of difficult-to-predict target detection of sample images and the imbalance in the number of sample images, and improves the accuracy of the model in target detection.

[0072] In one possible implementation, the loss weight corresponding to the detection target is positively correlated with the inverse of the number of sample images that include the detection target in the first data set; or, the loss weight corresponding to the detection target is determined based on the quantity interval in which the number of sample images that include the detection target in the first data set lies, wherein different quantity intervals correspond to different loss weights.

[0073] For detection targets with a large number of sample images, the loss weight of the detection target can be set to a smaller value, otherwise a larger value. For multi-detection target tasks, if the loss weight of each detection target is the same, it only plays the role of adjusting the loss multiple. Adjusting the multiple has no effect on training. Adjusting the gradient is the key to training. Therefore, in order to adjust the imbalance of the number of sample images of different detection targets, the loss weight corresponding to each detection target must be different. At this time, the loss weight is an array, in which each element represents the weight corresponding to each detection target.

[0074] The loss weight corresponding to the detection target is positively correlated with the inverse of the number of sample images including the detection target in the first data set. For example, if the number of sample images of the detection target cat is 3000, then the loss weight of the detection target cat is positively correlated with 1 / 3000.

[0075] Alternatively, the loss weight corresponding to the detection target is determined according to the quantity interval of the number of sample images that include the detection target in the first data set. At this time, it can be used to deal with the situation where the number of sample images is extremely unbalanced, such as when the sample size of a certain category is between 1-100, the α value is assigned to 0.2, between 100-1000, it is assigned to 0.5, between 1000-5000, it is assigned to 0.8, and so on.

[0076] In an embodiment of the present application, the loss weight of each detection target is determined according to the number of sample images corresponding to the detection target or the number interval in which the number of sample images lies, so that the loss weight corresponding to each detection target is different. The loss weight plays the role of adjusting the gradient, which can alleviate the problem of imbalance in the number of sample images corresponding to the detection target and improve the accuracy of the model in target detection.

[0077] Target detection device

[0078] Figure 3 is a schematic diagram of a target detection device provided by an embodiment of the present application, such as Figure 3 As shown, the target detection device 300 may include: a data collection module 302 , an acquisition module 304 , a first training module 306 , a second training module 308 and a target detection module 310 .

[0079] The data acquisition module 302 is used to acquire a first data set and a second data set, wherein the first data set includes a plurality of labeled sample images, and the second data set includes a plurality of unlabeled sample images.

[0080] The multiple sample images included in the first data set and the second data set may be collected by means of a camera device, an infrared sensor, etc., and this embodiment of the present application does not impose any limitation on this.

[0081] The first data set includes multiple labeled sample images. The collected sample images are manually annotated according to the annotation specifications, and a label is assigned to each sample image. The label contains the category information and location coordinates represented by the sample image. After manual annotation, the first data set is divided into a training set, a validation set, and a test set according to a certain ratio.

[0082] The second dataset includes multiple unlabeled sample images, which are not annotated for the collected sample images. They usually contain original images but lack clear labels to identify the categories and location coordinates of the sample images.

[0083] The acquisition module 304 is used to acquire a first target detection model.

[0084] Before training, the model structure is improved so that the first target detection model includes at least two detection heads, different detection heads correspond to different detection target groups, the detection target groups include at least two different categories of detection targets, and the detection heads are used to detect the detection targets included in the corresponding detection target groups. For example, the first target detection model includes two detection heads, the detection targets included in the detection target group corresponding to the first detection head are cats and feet, and the detection targets included in the detection target group corresponding to the second detection head are small passenger cars and large trucks.

[0085] The first target detection model can be obtained by training based on the YOLOV7 deep learning model. First, a public target detection data set, such as COCO and VOC data sets, is obtained. Then, an improved YOLOV7 is trained based on the public data set to obtain the first target detection model. It should be noted that YOLOV7 is only exemplary, and other deep learning models with target detection capabilities can also be selected, which is not limited in the embodiments of the present application.

[0086] The first training module 306 is used to train the first object detection model using the first data set to obtain a second object detection model.

[0087] The first data set includes multiple labeled sample images. Training the first target detection model using the first data set is supervised training. The first target detection model learns the relationship between input and output through known input sample images and corresponding output labels, thereby obtaining a second target detection model that can perform target detection on sample images.

[0088] The second training module 308 is used to train the second object detection model according to the first data set and the second data set to obtain a third object detection model.

[0089] The first data set includes labeled sample images, and the second data set includes unlabeled sample images. The first data set and the second data set are jointly used to train the model, namely semi-supervised training. In the case of a limited labeled first data set, the second data set including unlabeled sample images can be used to enhance the generalization ability of the model.

[0090] The object detection module 310 is used to perform object detection on the image to be detected by using the third object detection model.

[0091] The third target detection model obtained after supervised training and semi-supervised training has target detection capability and strong generalization ability, and can be used for target detection on images to be detected.

[0092] In the embodiment of the present application, a first data set including labeled sample images and a second data set of unlabeled sample images are acquired through the data acquisition module 302, and the acquisition module 304 acquires a first target detection model, the first target detection model includes at least two detection heads, different detection heads correspond to different detection target groups, the detection target groups include at least two different categories of detection targets, and the detection heads are used to detect the detection targets included in the corresponding detection target groups. The first training module 306 trains the first target detection model through the first data set to obtain the second target detection model, and then the second training model 308 trains the second target detection model according to the first data set and the second data set to obtain the third target detection model. The target detection module 310 performs target detection on the image to be detected through the third target detection model. The first target detection model includes at least two detection heads, and each detection head corresponds to a different detection target group. Each detection head only needs to detect the detection targets included in the corresponding detection target group. Different detection targets do not interfere with each other, which improves the accuracy of the model in target detection.

[0093] Electronic devices

[0094] Figure 4 This is a schematic block diagram of an electronic device provided in an embodiment of the present application. The specific embodiment of the present application does not limit the specific implementation of the electronic device. Figure 4 As shown, the electronic device 400 may include: a processor 402, a communications interface 404, a memory 406, and a communications bus 408. Among them:

[0095] The processor 402 , the communication interface 404 , and the memory 406 communicate with each other via a communication bus 408 .

[0096] The communication interface 404 is used to communicate with other electronic devices or servers.

[0097] The processor 402 is used to execute the program 410, and specifically can execute the relevant steps in any of the aforementioned target detection method embodiments.

[0098] Specifically, the program 410 may include program codes, which include computer operation instructions.

[0099] The processor 402 may be a CPU, or an application specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0100] The memory 406 is used to store the program 410. The memory 406 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0101] The program 410 may be specifically used to enable the processor 402 to execute the target detection method in any of the aforementioned embodiments.

[0102] The specific implementation of each step in program 410 can refer to the corresponding description of the corresponding steps and units in any of the above-mentioned target detection method embodiments, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above-mentioned method embodiments, which will not be repeated here.

[0103] The electronic device of the embodiment of the present application obtains a first data set including a labeled sample image and a second data set of unlabeled sample images. The first target detection model includes at least two detection heads, and different detection heads correspond to different detection target groups. The detection target groups include at least two different categories of detection targets. The detection heads are used to detect the detection targets included in the corresponding detection target groups. The first target detection model is trained with the first data set to obtain the second target detection model. Then, the second target detection model is trained based on the first data set and the second data set to obtain the third target detection model. The third target detection model is used to perform target detection on the image to be detected. The first target detection model includes at least two detection heads, and each detection head corresponds to a different detection target group. Each detection head only needs to detect the detection targets included in the corresponding detection target group. Different detection targets do not interfere with each other, which improves the accuracy of the model in performing target detection.

[0104] Storage Media

[0105] In this embodiment, a computer-readable storage medium is provided, which stores instructions for causing a machine to execute the target detection method as described herein. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0106] In this case, the program code read from the storage medium itself can implement the function of any one of the above embodiments, so the program code and the storage medium storing the program code constitute part of the present application.

[0107] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.

[0108] It should be noted that in the description of this application, the terms "first" and "second" are only used to facilitate the description of different components or names, and cannot be understood as indicating or implying a sequential relationship, relative importance, or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0109] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0110] It should be noted that although the specific embodiments of the present application are described in detail in conjunction with the accompanying drawings, this should not be construed as limiting the scope of protection of the present application. Within the scope described in the claims, various modifications and variations that can be made by those skilled in the art without creative work still fall within the scope of protection of the present application.

[0111] The examples of the embodiments of the present application are intended to concisely illustrate the technical features of the embodiments of the present application so that those skilled in the art can intuitively understand the technical features of the embodiments of the present application, and are not intended to be improper limitations of the embodiments of the present application.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A target detection method, characterized in that: The method comprises: Acquire a first data set and a second data set, wherein the first data set includes a plurality of labeled sample images, and the second data set includes a plurality of unlabeled sample images; Acquire a first target detection model, wherein the first target detection model includes at least two detection heads, different detection heads correspond to different detection target groups, the detection target groups include at least two different categories of detection targets, and the detection heads are used to detect the detection targets included in the corresponding detection target groups; Training the first target detection model using the first data set to obtain a second target detection model; Training the second object detection model according to the first data set and the second data set to obtain a third object detection model; Object detection is performed on the image to be detected by using the third object detection model.

2. The method according to claim 1, characterized in that The step of training the first target detection model by using the first data set to obtain a second target detection model includes: According to the detection target group corresponding to the detection head and the label of the sample image in the first data set, the sample image is subjected to feature extraction at different downsampling multiples to obtain at least two labeled feature maps, and the feature map whose corresponding label is located in the target detection group corresponding to the detection head is determined as the training data corresponding to the detection head, wherein different feature maps correspond to different resolutions; The detection head is trained using the training data corresponding to the detection head.

3. The method according to claim 2, characterized in that The detection head includes at least two sub-detection heads, and different sub-detection heads correspond to different resolutions; The step of training the detection head by using the training data corresponding to the detection head comprises: According to the resolution corresponding to the sub-detection head, the training data with the same resolution corresponding to the sub-detection head is input into the sub-detection head for target detection, so as to train the sub-detection head.

4. The method according to claim 1, characterized in that: The step of training the second target detection model according to the first data set and the second data set to obtain a third target detection model includes: Determining the second target detection model as a model to be trained; Performing target detection on sample images in the second data set by using the model to be trained to obtain pseudo labels for the sample images in the second data set; Generate an intermediate data set including sample images and corresponding pseudo labels in the second data set; Training the second object detection model using the first data set and the intermediate data set to obtain an intermediate model; If the performance of the intermediate model no longer increases, determining the intermediate model as the third object detection model; If the performance of the intermediate model is increasing, the intermediate model is determined as the model to be trained, and the target detection is performed on the sample images in the second data set by using the model to be trained.

5. The method according to claim 1, characterized in that The training of the first target detection model using the first data set includes: Determining a category weight of a corresponding detection target according to the number of detection targets included in the sample images in the first data set, wherein the category weight of the detection target is negatively correlated with the number of the detection targets included in the first data set; The sample weight of the sample image is calculated according to the category weight of the detection target included in the sample image in the first data set, and the sample image in the first data set is loaded according to the sample weight of the sample image, and the first target detection model is trained, wherein the number of times the sample image is loaded is positively correlated with the sample weight of the sample image.

6. The method according to claim 1, characterized in that The loss function of the first target detection model includes: FL α =-α(1-p t ) γ log(p t ) FL α A loss function for characterizing the first target detection model for a sample image including a detection target in a current detection target group; It is used to characterize the result of target detection performed by the first target detection model on the sample image including the detection targets in the current detection target group, and the result of the target detection is used to indicate the probability that the sample image includes each detection target in the current detection target group; y is used to characterize the label corresponding to the detection target included in the sample image in the first data set, and α is used to characterize the loss weight corresponding to the current detection target group; γ is used to characterize the attention corresponding to the current detection target group, and γ is positively correlated with the detection difficulty value of the detection target included in the current detection target group.

7. The method according to any one of claims 1 to 6, characterized in that: The loss weight corresponding to the detection target is positively correlated with the inverse of the number of sample images including the detection target in the first data set; or, The loss weight corresponding to the detection target is determined according to the quantity interval of the sample images including the detection target in the first data set, wherein different quantity intervals correspond to different loss weights.

8. A target detection device, characterized in that: The device comprises: A data acquisition module, used to acquire a first data set and a second data set, wherein the first data set includes a plurality of labeled sample images, and the second data set includes a plurality of unlabeled sample images; An acquisition module, used to acquire a first target detection model, wherein the first target detection model includes at least two detection heads, different detection heads correspond to different detection target groups, the detection target group includes at least two different categories of detection targets, and the detection head is used to detect the detection targets included in the corresponding detection target group; A first training module, used to train the first target detection model using the first data set to obtain a second target detection model; A second training module, used for training the second object detection model according to the first data set and the second data set to obtain a third object detection model; The target detection module is used to perform target detection on the image to be detected by using the third target detection model.

9. An electronic device, characterized in that: include: A processor, a communication interface, a memory and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the target detection method according to any one of claims 1 to 7.

10. A computer storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the target detection method according to any one of claims 1 to 7 is implemented.

Images