Model training method, target detection method, device, equipment and storage medium

Abstract

The embodiments of the present application disclose a model training method, a target detection method, an apparatus, a device and a storage medium, wherein the method comprises: obtaining a first sample with an instance-level label and a second sample with an image-level label; the second sample is obtained based on the instance-level label of the first sample; determining a pseudo-label of sample data in the second sample through a pre-trained target detection model; determining an original detection loss of the target detection model based on the instance-level label of the sample data in the first sample; determining a classification enhancement loss of the target detection model based on the pseudo-label of the sample data in the second sample; and training the target detection model using the first sample and the second sample based on the original detection loss and the classification enhancement loss.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and is related to but not limited to model training methods, target detection methods, devices, equipment and storage media. Background Art

[0002] At present, mainstream object detection technologies are all applied in scenarios with relatively balanced data. However, in reality, as the number of categories to be detected increases, the categories will naturally present a long-tail distribution. This serious category imbalance problem will greatly affect the performance of the object detector.

[0003] For long-tail object detection, the mainstream method now is to use loss reweighting and data resampling, but these methods are still performed under limited data sets. For some minority categories of images, the amount of data may be very small, which makes neural networks, which are highly data-dependent, unable to show good generalization performance on these minority categories. Summary of the invention

[0004] Embodiments of the present application provide a model training method, a target detection method, an apparatus, a device, and a storage medium.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a model training method, comprising:

[0007] Obtain a first sample with an instance-level label and a second sample with an image-level label; the second sample is obtained based on the instance-level label of the first sample; determine the pseudo-label of the sample data in the second sample through a pre-trained target detection model; determine the original detection loss of the target detection model based on the instance-level label of the sample data in the first sample; determine the classification enhancement loss of the target detection model based on the pseudo-label of the sample data in the second sample; and train the target detection model using the first sample and the second sample based on the original detection loss and the classification enhancement loss.

[0008] In some possible embodiments, the obtaining of the first sample with the instance-level label and the second sample with the image-level label includes: obtaining the first sample with the instance-level label from the sample training set; determining at least one category included in the first sample based on the instance-level label; and obtaining a second sample corresponding to the category based on the at least one category to obtain the second sample with the image-level label. In this way, based on the various categories included in the first sample, obtaining rich weakly annotated second samples as additional data can improve the classification ability of the object detector.

[0009] In some possible embodiments, the at least one category is the category to which all sample data in the first sample belongs, and the all sample data includes normal data and long-tail data; based on the at least one category, obtaining the second sample corresponding to the corresponding category to obtain the second sample with an image-level label includes: obtaining the second sample corresponding to each category through a preset image data set to obtain the second sample with an image-level label. In this way, additional second samples with image-level labels, i.e., weak labels, can be quickly obtained through the preset image data set. Thus, the classification ability of the target detection model for the tail category is enhanced through the second sample annotated at the image level.

[0010] In some possible embodiments, the method of determining the pseudo-label of the sample data in the second sample by using a pre-trained target detection model includes: inputting the second sample into the pre-trained target detection model to obtain a set of candidate boxes in the sample data in the second sample; determining the classification confidence of each candidate box in the candidate box set under the image-level label of the corresponding sample data; sorting the classification confidence of each candidate box in the candidate box set to obtain the target box with the highest classification confidence score; and generating the pseudo-label of the sample data in the second sample based on the target box with the highest classification confidence score. In this way, based on the acquired image-level label of the second sample, the candidate boxes detected by the target detection model are sorted using the confidence of the category to which the sample data in the second sample belongs, and the target box with the highest confidence score is selected to generate an accurate pseudo-label for the sample data in the corresponding second sample, thereby greatly improving the performance of the model.

[0011] In some possible embodiments, the target detection model includes a first branch network for performing a positioning task and a second branch network for performing a classification task; the determining of the original detection loss of the target detection model based on the instance-level label of the sample data in the first sample includes: determining the original detection loss of the first sample through the first branch network and the second branch network based on the instance-level label of the sample data in the first sample; the determining of the classification enhancement loss of the target detection model based on the pseudo-label of the sample data in the second sample includes: determining the classification enhancement loss of the second sample through the second branch network of the target detection model based on the pseudo-label of the sample data in the second sample. In this way, the positioning task and the classification task of target detection are decoupled through the first branch network and the second branch network, so that only the acquired second sample, i.e., the additional data, is used to enhance the classification part of the target detection model, thereby being able to efficiently utilize a large amount of additional data.

[0012] In some possible embodiments, the training of the target detection model using the first sample and the second sample based on the original detection loss and the classification enhancement loss includes: determining the task balance coefficient of the classification task and the positioning task; based on the task balance coefficient, linearly summing the original detection loss and the classification enhancement loss to obtain the optimized target loss of the target detection model; and training the target detection model using the first sample and the second sample under the supervision of the optimized target loss. In this way, by introducing a suitable task balance coefficient to balance the original detection loss and the classification enhancement loss, the detection accuracy of the target detection model is further improved.

[0013] In some possible embodiments, the target detection model further includes a region candidate network, and the determining of the original detection loss of the first sample through the first branch network and the second branch network based on the instance-level label of the sample data in the first sample includes: determining the candidate box set and the region candidate loss of the sample data in the first sample through the region candidate network; determining the classification loss of the first sample through the second branch network based on the instance-level label of the sample data in the first sample and the candidate box set; determining the regression loss of the first sample through the first branch network based on the instance-level label of the sample data in the first sample and the forward candidate box containing the target in the candidate box set; determining the original detection loss of the first sample through the first branch network and the second branch network based on the region candidate loss, the classification loss and the regression loss. In this way, the region candidate loss generated by the first sample through the region candidate network, the classification loss generated by the second branch network and the regression loss generated by the first branch network are calculated respectively, and the original detection loss of the final target detection model is determined, thereby improving the accuracy of target detection.

[0014] In some possible embodiments, the determining the original detection loss of the first sample through the first branch network and the second branch network based on the instance-level label of the sample data in the first sample also includes: determining the target estimation loss of the first sample through the first branch network based on the instance-level label of the sample data in the first sample and the candidate box set; determining the original detection loss of the first sample through the first branch network and the second branch network based on the region candidate loss, the classification loss, the target estimation loss and the regression loss. In this way, introducing an additional target estimation loss to estimate the objectivity of the candidate box can alleviate the imbalance problem between the foreground and background in the image and increase the average accuracy of target detection. At the same time, adding a target estimation task to the first branch network can achieve higher performance improvement.

[0015] In some possible embodiments, the pseudo label of the second sample also includes the position information of the target box in the second sample, and the target box is the candidate box corresponding to the highest classification confidence in the second sample; the training of the target detection model using the first sample and the second sample based on the original detection loss and the classification enhancement loss also includes: before each iteration of training, based on the position information of the target box in the sample data in the second sample, determining the target area of ​​the sample data corresponding to the second sample; fusing the target area of ​​at least one sample data in the second sample to obtain a synthetic sample image; based on the original detection loss and the classification enhancement loss, using the first sample, the second sample and the synthetic sample image to train the target detection model. In this way, the context around the target box is randomly cropped to obtain the regional image corresponding to the second sample, and then the synthetic sample image is constructed, and the synthetic sample image is combined with the training to avoid noise from the cluttered background and improve the efficiency of target detection.

[0016] In some possible embodiments, the step of fusing the target region of the sample data in at least one of the second samples to obtain a synthetic sample image includes: performing data enhancement processing on the target region of the sample data in the second sample; the data enhancement processing includes at least one of the following operations: random scaling, cropping, or mosaic; fusing at least one of the enhanced target regions to obtain a synthetic sample image. In this way, the image of the region where the target box in the sample data is located is subjected to data enhancement processing such as random scaling, cropping, mosaic, etc., and then spliced ​​to form a synthetic sample image, which can improve the robustness of model training.

[0017] In some possible embodiments, the first branch network and the second branch network include the same two-layer fully connected layer structure, and / or the first branch network and the second branch network are trained in parallel. In this way, by setting the two branch networks to use the same two-layer fully connected layer structure but not sharing parameters, the classification task and the positioning task are decoupled, so that the additional second samples can be effectively used to enhance the classification ability of the target detection model.

[0018] In a second aspect, an embodiment of the present application provides a target detection method, comprising: obtaining a target image to be detected; recognizing the target image through a first branch network of a trained target detection model to obtain a detection frame of the target in the target image and a corresponding objective evaluation score; recognizing the target image through a second branch network of the target detection model to obtain a category of the target in the target image; and determining a target detection result of the target image based on the detection frame of the target, the objective evaluation score and the category of the target.

[0019] In a third aspect, an embodiment of the present application provides a model training device, the device comprising a first acquisition module, a first determination module, a second determination module, a third determination module and a training module, wherein:

[0020] The first acquisition module is used to acquire a first sample with an instance-level label and a second sample with an image-level label; the second sample is acquired based on the instance-level label of the first sample;

[0021] The first determination module is used to determine the pseudo labels of the sample data in the second sample by using a pre-trained target detection model;

[0022] The second determination module is used to determine the original detection loss of the target detection model based on the instance-level label of the sample data in the first sample;

[0023] The third determination module is used to determine the classification enhancement loss of the object detection model based on the pseudo labels of the sample data in the second sample;

[0024] The training module is used to train the target detection model using the first sample and the second sample based on the original detection loss and the classification enhancement loss.

[0025] In a fourth aspect, an embodiment of the present application provides a target detection device, the device comprising a second acquisition module, a first recognition module, a second recognition module and a fourth determination module, wherein:

[0026] The second acquisition module is used to acquire the target image to be detected;

[0027] The first recognition module is used to recognize the target image through a first branch network of a trained target detection model to obtain a detection frame of the target in the target image and a corresponding objective evaluation score;

[0028] The second recognition module is used to recognize the target image through the second branch network of the target detection model to obtain the category of the target in the target image;

[0029] The fourth determination module is used to determine the target detection result of the target image based on the detection frame of the target, the objective evaluation score and the category of the target.

[0030] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and when the processor executes the program, the steps in the above-mentioned target detection method are implemented.

[0031] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps in the above-mentioned target detection method when executed by a processor.

[0032] The beneficial effects brought about by the technical solution provided by the embodiment of the present application include at least: in the embodiment of the present application, based on the instance-level label of the first sample, a second sample with an image-level label is obtained, and a pseudo-label of the second sample is generated through a pre-trained target network model, and finally, by determining the original detection loss corresponding to the first sample and the classification enhancement loss corresponding to the second sample, the collected second sample is jointly trained with the original first sample in a classification enhancement manner. The embodiment of the present application improves the performance of long-tail target detection by using additional data with weak labels, i.e., image-level labels, which enables the target detection model to achieve better performance while being less prone to overfitting. At the same time, the embodiment of the present application only uses the second sample to enhance the classification part of the model, which can efficiently utilize a large amount of additional data. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0034] Figure 1 A flowchart of a model training method provided in an embodiment of the present application;

[0035] Figure 2 A flowchart of a model training method provided in an embodiment of the present application;

[0036] Figure 3A A flowchart of a model training method provided in an embodiment of the present application;

[0037] Figure 3B Comparison of classification confidence scores of the target detection model provided in the embodiment of the present application on different data sets;

[0038] Figure 4 A flowchart of a model training method provided in an embodiment of the present application;

[0039] Figure 5A A schematic diagram of a target detection method according to an embodiment of the present invention;

[0040] Figure 5B A logical flow chart of the target detection model training method provided in the embodiment of the present application;

[0041] Figure 5C A system architecture diagram of the target detection model training method provided in an embodiment of the present application;

[0042] Figure 6 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0043] Figure 7 A schematic diagram of the structure of a target detection device provided in an embodiment of the present application;

[0044] Figure 8 A hardware entity schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. The following embodiments are used to illustrate the present application, but are not intended to limit the scope of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0046] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0047] It should be pointed out that the terms "first\second\third" involved in the embodiments of the present application are merely to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in the preset order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0048] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as the general understanding of those skilled in the art in the field to which the embodiments of the present application belong. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0049] The solution provided in the embodiment of the present application relates to the field of deep learning technology. In order to facilitate the understanding of the solution in the embodiment of the present application, the terms involved in the relevant technology are first briefly explained:

[0050] Long-tail scenarios: mainly refer to a wide variety of scenarios with low probability of occurrence or sudden situations, such as vehicles running red lights, pedestrians crossing the road, intersections with damaged traffic lights, and vehicles parked illegally on the roadside.

[0051] Object detection is to accurately find the location of an object in a given image (detection box annotation) and mark the category of the object. The problem that object detection needs to solve is where the object is and what it is.

[0052] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0053] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0054] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0055] Deep learning: It is to learn the inherent laws and representation levels of sample data. The information obtained in the learning process is very helpful for interpreting data such as text, images and sounds. The ultimate goal is to enable machines to have analytical learning capabilities like humans and be able to recognize data such as text, images and sounds.

[0056] Computer vision is an important direction of artificial intelligence technology. In computer vision processing, it is usually necessary to detect targets in images or videos, such as pedestrians, objects, etc. Large-scale long-tail data target detection has important applications in many fields, such as abnormal object detection, abnormal behavior detection, and emergency alarm.

[0057] Long-tail object detection: Long-tail object detection aims to train object detectors on extremely unbalanced training sets and achieve good results on balanced test sets. In this case, the model is easily biased towards frequent categories and thus performs poorly on rare categories.

[0058] Weakly supervised object detection: Object detectors are trained based on images with image-level labels. The accuracy of weakly supervised object detectors lags far behind supervised object detectors, especially in more complex scenes.

[0059] Semi-supervised object detection: Semi-supervised object detection occurs when the amount of labeled data is relatively small and the amount of unlabeled data is relatively large. Inherited from semi-supervised image classification, two widely used approaches for semi-supervised object detection are consistency-based and pseudo-label-based. Pseudo-label-based approaches have proven to be a good solution for semi-supervised object detection. However, due to poor performance in the context of long-tail detection, predicting pseudo-labels for rare categories becomes a disaster.

[0060] Weakly Semi-Supervised Object Detection: Usually there are two datasets for weakly semi-supervised object detection: one contains a dataset with instance-level annotations, and the other contains a dataset with only image-level annotations.

[0061] Different from detection datasets that require instance-level annotations such as detection boxes, classification data is very easy to obtain. For example, classification data can be quickly obtained through online search engines; there are also a large number of classification datasets in the community, such as ImageNet, etc. Therefore, this part of extra data can be used to improve the performance of object detectors in long-tail situations.

[0062] The existing method of utilizing additional data is mainly semi-supervised object detection (SSOD), which mainly uses a calibrated pseudo-label method to allow the model to predict and generate labels for additional data, and then mixes the manually labeled data with the labels predicted by the model for training. During the research process, it was found that in the long-tail case, since the target detector performs poorly on a few categories, this semi-supervised learning method does not perform very well. This is mainly reflected in the fact that the labels predicted by the model contain a lot of noise, so that training based on the generated labels will reduce the performance of the model. On the other hand, since the amount of additional data is often much larger than the amount of labeled data, the effectiveness of data utilization is also a very important issue to consider.

[0063] The embodiment of the present application provides a model training method, which is applied to an electronic device or a server. The electronic device includes but is not limited to a mobile phone, a laptop computer, a tablet computer and a handheld Internet device, a multimedia device, a streaming media device, a mobile Internet device, a wearable device or other types of devices. The function implemented by the method can be implemented by calling the program code by the processor in the electronic device. Of course, the program code can be stored in a computer storage medium. It can be seen that the electronic device includes at least a processor and a storage medium. The processor can be used to process the process of generating distillation data, and the memory can be used to store the intermediate data required in the process of generating distillation data and the generated target distillation data.

[0064] Figure 1 A flow chart of the model training method provided in the embodiment of the present application is as follows: Figure 1 As shown, the method comprises at least the following steps:

[0065] Step S110, obtaining a first sample with an instance-level label and a second sample with an image-level label;

[0066] Here, the second sample is obtained based on the instance-level label of the first sample; during training, the first sample with the instance-level label is used as a fully supervised dataset, and the second sample with the image-level label is used as a semi-supervised dataset.

[0067] It should be noted that the first sample with instance-level labels, that is, the detection dataset D generated by instance-level annotation f ={(x i ,y i )}, called the fully labeled dataset, where each first sample image x i Contains instance y i ={(b,c)}, examples include candidate box position b = (b cx ,b cy,b w ,b h ) and one-hot class label c∈{0,1} |C| , |C| is the total number of all categories. That is, the first sample is marked with the candidate boxes detected in the image and the corresponding categories.

[0068] For example, if an image includes different animals such as cats and dogs, the result of instance-level annotation is candidate box 1 and category "cat" where the cat is located, and candidate box 2 and category "dog" where the dog is located. If an image includes multiple cats, each cat is an instance, and the result of instance-level annotation is the candidate box and category "cat" where each cat is located.

[0069] The second sample with image-level labels, that is, the classification dataset with only labeled categories, is called the weakly labeled dataset D w ={(x i ,c)}, each second sample image x i Use Mark, where Indicates the second sample image x i Whether it contains at least one instance of a category. In other words, the second sample is only labeled with the category label of the category to which the image belongs, that is, a second sample is only associated with one category label, representing the only category to which it belongs. For example, the result of image-level annotation is that a dog image is assigned a category label "dog", and a cat image is assigned a category label "cat".

[0070] Step S120, determining a pseudo label of the sample data in the second sample by using a pre-trained target detection model;

[0071] Here, the target detection model may be, for example, a convolutional neural network (CNN), a regional convolutional neural network (R-CNN), a fast convolutional neural network (Fast R-CNN), or a faster convolutional neural network (Faster R-CNN). The embodiment of the present application does not limit the specific network structure of the target detection model.

[0072] The second sample is input into the pre-trained target detection network for processing, and a pseudo label of the sample data in the second sample is obtained, and the pseudo label includes the location area of ​​the target in the second sample, i.e., the target frame, and the classification probability. The location area is the range of the detected target frame (center point coordinates and the length and width of the target frame), and the classification probability is the confidence score of the target in different categories.

[0073] Step S130, determining the original detection loss of the object detection model based on the instance-level label of the sample data in the first sample;

[0074] Here, for the first sample, the target detection model is trained according to the standard detection training method, and the original detection loss is calculated based on the detection result output by the target detection model and the instance-level label of the first sample.

[0075] In some embodiments, the original detection loss includes the regression loss generated in performing the positioning task and the classification loss generated in performing the classification task; wherein the positioning task refers to determining the detection box position of the target in the sample image, that is, where the target is; the classification task refers to determining the category to which the target in the sample image belongs, that is, what the target is.

[0076] In some embodiments, the original detection loss also includes the region candidate loss generated in the process of generating the candidate box set and the target estimation loss in the process of performing the objectivity estimation task. The target estimation loss can be used to multiply the objectivity score by the classification score during the reasoning process to obtain the final estimated probability.

[0077] Step S140, determining the classification enhancement loss of the object detection model based on the pseudo labels of the sample data in the second sample;

[0078] Here, unlike the traditional semi-supervised target detection, the second sample with pseudo labels and the first sample with instance-level labels are processed in the same way, such as generating a set of candidate boxes, sampling candidate boxes, calculating regression loss and classification loss, etc. The embodiment of the present application only uses the second sample and their respective pseudo labels to enhance the classification ability of the target detection model.

[0079] In implementation, the classification enhancement loss of the second sample through the target detection model can be calculated by the following formula (1):

[0080] L w (x w ,q w )=L cls (x w ,a g ,q w ) (1);

[0081] Among them, L w (x w ,q w ) is the classification enhancement loss of the second sample through the target detection model, x w is the second sample, q w is the second sample x generated by the pre-trained object detection network w The pseudo label of gis the second sample x w The target box where the target is located.

[0082] It can be seen that, in contrast to the traditional classification loss in target detection, the negative candidate box (i.e., the background candidate box, excluding the target) is not used in the calculation of the classification enhancement loss in the embodiment of the present application, and only the target box is used as the positive candidate box (including the candidate box of the target).

[0083] Step S150: Based on the original detection loss and the classification enhancement loss, the target detection model is trained using the first sample and the second sample.

[0084] Here, the original detection loss and the classification enhancement loss are combined, and the target detection model is jointly trained using the first sample and the second sample.

[0085] In an embodiment of the present application, based on the instance-level label of the first sample, a second sample with an image-level label is obtained, and a pseudo-label of the second sample is generated through a pre-trained target network model. Finally, by determining the original detection loss corresponding to the first sample and the classification enhancement loss corresponding to the second sample, the collected second sample is jointly trained with the original first sample in a classification enhancement manner. The embodiment of the present application improves the performance of long-tail target detection by using additional data with weak labels, i.e., image-level labels, which enables the target detection model to achieve better performance while being less prone to overfitting. At the same time, the embodiment of the present application only uses the second sample to enhance the classification part of the model, which can efficiently utilize a large amount of additional data.

[0086] Figure 2 A flow chart of the model training method provided in the embodiment of the present application is as follows: Figure 2 As shown, the method comprises at least the following steps:

[0087] Step S210, obtaining a first sample with an instance-level label from a sample training set;

[0088] Here, the sample training set can be a large instance segmentation dataset (Large Vocabulary Instance Segmentation, LVIS), a large vocabulary dataset for target detection and instance segmentation, which contains 1203 categories and follows a long-tail distribution. The sample training set can also be a common object dataset (Common Objects in Contex, COOC) and the like. During model training, the first sample with instance-level labels can be directly used as a detection dataset for the target detector.

[0089] Step S220, determining at least one category included in the first sample based on the instance-level label;

[0090] Here, the instance-level label of the sample data in the first sample includes the candidate box position of the instance and the category label of the instance, so that all categories included in the first sample can be counted. It is worth noting that all categories included in the first sample include normal categories and sparse long-tail categories.

[0091] Step S230, based on the at least one category, obtaining a second sample corresponding to the category, to obtain the second sample with an image-level label;

[0092] Here, the second sample with the image-level label is additionally acquired classification data.

[0093] In some implementations, all categories existing in the existing data, i.e., the first sample, are first counted, and then a certain amount of additional data is collected for each category as the second sample. Thus, rich weakly annotated second samples can be obtained based on each category as additional data to improve the fully supervised object detector and enhance the classification capability of the object detector.

[0094] In other embodiments, rare categories or tail categories present in the first sample are screened out, and a certain amount of additional data is collected for each tail category as the second sample, thereby enhancing the classification capability of the object detection model for the tail category through the image-level annotated second sample.

[0095] It should be noted that, unlike sample datasets that require instance-level annotations such as candidate boxes, classification data with only image-level labels is very easy to obtain. For example, you can quickly obtain second samples through a web search engine. There are also a large number of image datasets in the community, such as ImageNet, which can directly collect corresponding classification images with image-level labels based on category names. Therefore, this additional data can be used to improve the performance of object detectors in long-tail situations.

[0096] In some implementations, the second sample corresponding to each of the categories is obtained through a search engine, an academic forum, a domain database, etc., to obtain the second sample with an image-level label. For example, Baidu search or Google search can be used to directly enter the field of the category to be collected, such as "cat", and crawler technology can be used to directly download and save the second sample of the corresponding category, such as an image containing a cat.

[0097] In some other embodiments, the second sample corresponding to the category is obtained through a preset image dataset to obtain the second sample with an image-level label. For example, the second sample is obtained through an image dataset that already exists in the community. In this case, the corresponding pictures can be collected directly according to the name (or synonym) of the category. The preset image dataset can be ImageNet-22k.

[0098] In some further implementations, the second sample corresponding to each of the categories is acquired through a search engine and a preset image data set to obtain the second sample with an image-level label.

[0099] Step S240, determining a pseudo label of the sample data in the second sample by using a pre-trained target detection model;

[0100] Step S250, determining the original detection loss of the object detection model based on the instance-level label of the sample data in the first sample;

[0101] Step S260, determining the classification enhancement loss of the object detection model based on the pseudo labels of the sample data in the second sample;

[0102] Step S270: Based on the original detection loss and the classification enhancement loss, the target detection model is trained using the first sample and the second sample.

[0103] In the embodiment of the present application, based on the categories included in the first sample, an additional second sample with an image-level label, i.e., a weak label, is obtained through a preset image data set. In this way, the first sample and the second sample are used to jointly train the target detection model, thereby improving the performance of long-tail target detection. The algorithm enables the target detection model to achieve better performance while being less prone to overfitting.

[0104] In some embodiments, the pseudo label of the second sample further includes position information of a target box in the second sample, and the target box is a candidate box corresponding to the highest classification confidence in the second sample; step S270 may include the following steps:

[0105] Before each iterative training, based on the position information of the target box in the sample data in the second sample, determine the target area of ​​the sample data in the second sample; perform data enhancement processing on the target area of ​​the sample data in the second sample; fuse the target area after at least one enhancement processing to obtain a synthetic sample image; based on the original detection loss and the classification enhancement loss, use the first sample, the second sample and the synthetic sample image to train the target detection model.

[0106] Here, the position information of the target frame in the second sample includes the coordinates of the center point of the target frame and the size of the target frame. In this way, the context around the target frame is randomly cropped to obtain the regional image corresponding to the second sample, and then a synthetic sample image is constructed. The synthetic sample image is combined with the synthetic sample image for training, which can avoid noise from cluttered backgrounds and improve the efficiency of target detection.

[0107] In some embodiments, the fusing of the target area of ​​sample data in at least one of the second samples to obtain a synthetic sample image includes: performing data enhancement processing on the target area of ​​sample data in the second sample; the data enhancement processing includes at least one of the following operations: random scaling, cropping or mosaic; fusing the target area after at least one enhancement processing to obtain a synthetic sample image.

[0108] Thus, before each iteration of training, for each image in the second sample, several regional images where the target box is located are randomly selected, and then randomly scaled, cropped, mosaiced, and other data enhancement processes are performed and then spliced ​​to form a synthetic sample image, which is trained together with other second samples. By default, four regional images can be set to be spliced ​​into one synthetic sample image.

[0109] based on Figure 1 , Figure 3A A flow chart of the model training method provided in the embodiment of the present application is as follows: Figure 3A As shown, the step S120 "determining the pseudo label of the second sample by using the pre-trained target detection model" includes at least the following steps:

[0110] Step S310, inputting the second sample into a pre-trained object detection model to obtain a candidate box set of sample data in the second sample;

[0111] Here, the sample data in the second sample is an image with an image-level label. By performing target detection on the second sample through a pre-trained target detection model, a candidate box set corresponding to the sample data in the second sample can be obtained; wherein the candidate box set includes all detected detection boxes in this image, namely candidate boxes.

[0112] Step S320, determining the classification confidence of the candidate boxes in the candidate box set under the image-level labels of the corresponding sample data;

[0113] For each candidate box, the pre-trained object detection network will simultaneously output the confidence of the corresponding candidate box in each category, that is, the classification confidence. Assuming that the second sample includes C N After a second sample Di is detected by the pre-trained target detection network, the M candidate boxes included in the second sample Di and the C of each candidate box are obtained.N scores {S1, S2, ..., S N}.

[0114] Step S330, sorting the classification confidences of the candidate boxes in the candidate box set to obtain the target box with the highest classification confidence score;

[0115] Here, still taking the example in step S320 as an example, the image-level label of the second sample Di corresponds to a specific category Ci, and the scores S of the M candidate boxes under category Ci are i By sorting, we can get the candidate box with the highest classification confidence score and use it as the target box.

[0116] It should be noted that the most classic method for generating pseudo labels for additional data is semi-supervised target detection, which usually uses a pre-trained target detection model to directly predict on the additional data and collects target boxes with high classification confidence scores to generate pseudo labels, thereby avoiding the collection of a large number of noisy labels. However, research has found that in long-tail scenarios, the classification confidence scores of target detection models for a few categories are very low. If a high score is used to select the target box, a sufficient number of pseudo labels will not be selected; conversely, if a low score is used to select the target box, the generated pseudo labels will contain a lot of noise.

[0117] Figure 3B The figure shows the comparison of classification confidence scores of the target detection model on different data sets. It can be seen that on the LVIS data set, the classification confidence score of the target detection model for long-tail categories (category index is around 1200) is very low.

[0118] In the embodiment of the present application, by providing the image-level label of the sample data in the second sample, the target detection model is used to sort the candidate boxes in the candidate box set obtained in step S310, thereby determining the target box with the highest classification confidence score in the corresponding second sample, and then obtaining the pseudo label of the sample data in the corresponding second sample. In other words, by only selecting the candidate box with the highest score relative to the image label to generate the pseudo label, the influence of other noises is overcome.

[0119] Step S340: Generate a pseudo label for the sample data in the second sample based on the target box with the highest classification confidence score.

[0120] Here, the center point coordinates, size, and category of the target box in the second sample are marked on the sample data of the second sample to form a pseudo label of the sample data in the second sample.

[0121] In some implementations, the pseudo labels of the sample data in the second sample may be generated by manual annotation, or by other annotation devices, which is not limited in this embodiment of the present application.

[0122] In some embodiments, before training the target detection model, pseudo labels for the sample data in the second sample are generated offline in advance, thereby reducing computational costs and providing scalability for large weakly labeled data.

[0123] In an embodiment of the present application, based on the acquired image-level label of the second sample, the confidence of the category to which the sample data in the second sample belongs is used to sort the candidate boxes detected by the target detection model, and the target box with the highest confidence score is selected to generate a pseudo-label for the sample data in the corresponding second sample. By utilizing the image-level label information of the sample data in the second sample, the influence of noise is overcome; at the same time, by utilizing the characteristic of the target detection model that can accurately sort the candidate boxes, accurate pseudo-labels for the sample data in the second sample can be generated, thereby greatly improving the performance of the model.

[0124] In some embodiments, the target detection model includes a first branch network for performing a positioning task and a second branch network for performing a classification task. Among them, the positioning task refers to determining the detection box position of the target in the sample image, that is, where the target is; the classification task refers to determining the category to which the target in the sample image belongs, that is, what the target is. The embodiment of the present application uses a faster regional convolutional network (Faster R-CNN) as the basic model, and uses a shared subnetwork in the head of R-CNN to perform classification tasks and positioning tasks. Therefore, the embodiment of the present application constructs a separate detection head (Detection Head) to decouple the two tasks, and each branch network performs one of the above tasks respectively.

[0125] Figure 4 A flow chart of the model training method provided in the embodiment of the present application is as follows: Figure 4 As shown, the method comprises at least the following steps:

[0126] Step S410, obtaining a first sample with an instance-level label and a second sample with an image-level label;

[0127] Here, the second sample is obtained based on the instance-level label of the first sample.

[0128] Step S420, determining a pseudo label of the sample data in the second sample by using a pre-trained target detection model;

[0129] Here, the implementation process of determining the pseudo label of the sample data in the second sample can refer to the above Figure 3AThe corresponding embodiments.

[0130] Step S430, determining the original detection loss of the first sample through the first branch network and the second branch network based on the instance-level label of the sample data in the first sample;

[0131] Here, the first sample is simultaneously input into the first branch network and the second branch network of the target detection model, and the original detection loss is obtained by calculating the losses of the first branch network and the second branch network in combination with the first sample.

[0132] The first branch network is a branch for performing positioning tasks. The loss calculated under this branch includes at least regression loss. Usually, loss functions such as Mean Absolute Error (MAE) and Mean Square Error (MSE) are used to optimize the model's positioning results for the target; the second branch network is a branch for performing classification tasks. The loss calculated under this branch includes at least classification loss, which can be calculated by the Cross Entropy Loss function.

[0133] Step S440, determining the classification enhancement loss of the second sample through the second branch network of the object detection model based on the pseudo label of the sample data in the second sample;

[0134] Here, the second sample is only input into the second branch network of the target detection model, and based on the pseudo label of the second sample, the classification enhancement loss of the first sample through the second branch network is determined. The classification enhancement loss is calculated in the same way as the above classification loss.

[0135] It should be noted that after generating the pseudo-labels of the sample data in the second sample, the traditional semi-supervised method inputs the second sample together with the first sample into the two branch networks of the target detection model, that is, the acquired additional data is also used as the original sample data for target detection. If the additional data is 10 times the original sample data, the increased computational effort will also be 10 times the original amount, so this method has certain limitations when applying large-scale additional data.

[0136] Considering that in the long-tail scenario, the target detection model mainly lacks classification ability. Therefore, the embodiment of the present application only inputs the second sample into the second branch network of the target detection model for detection, that is, only the second sample is used to enhance the classification ability of the target detection model, while avoiding other unnecessary parts.

[0137] Step S450, determining a task balance coefficient between the classification task and the positioning task;

[0138] Here, the task balance coefficient is used to balance the origin detection loss and the additional classification loss. If the value of the task balance coefficient is too small, the object detection model cannot effectively learn from the second sample. A larger value of the task balance coefficient will damage the balance between the two tasks and the balance between foreground samples and background samples. The task balance coefficient is usually set to 0.1 to 0.2 based on experience.

[0139] Step S460, based on the task balance coefficient, linearly summing the original detection loss and the classification enhancement loss to obtain an optimized target loss of the target detection model;

[0140] Here, the optimized target loss of the target detection model is calculated by the following formula (2):

[0141] L=L f (x f ,q f )+αL w (x w ,q w ) (2);

[0142] Among them, L is the optimization target loss, L f (x f ,q f ) is the original detection loss of the first sample through the target detection model, L w (x w ,q w ) is the classification enhancement loss of the second sample through the target detection model, and α is the task balance coefficient.

[0143] Step S470: training the target detection model using the first sample and the second sample under the supervision of the optimized target loss.

[0144] In the embodiment of the present application, the first sample is input into the first branch network and the second branch network of the target detection model at the same time, and the second sample is only input into the second branch network of the target detection model, so that only the acquired second sample, i.e., the additional data, is used to enhance the classification part of the target detection model, thereby being able to efficiently utilize a large amount of additional data. At the same time, by introducing a suitable task balance coefficient to balance the original detection loss and the classification enhancement loss, the detection accuracy of the target detection model is further improved.

[0145] In some embodiments, the first branch network and the second branch network include the same two-layer fully connected layer structure, and / or the first branch network and the second branch network are trained in parallel.

[0146] In this way, by setting the two branch networks to use the same two-layer fully connected layer structure but not sharing parameters, the classification task and the positioning task are decoupled, making it easier to effectively use the additional second sample to enhance the classification ability of the target detection model.

[0147] In some embodiments, the target detection model further includes a region proposal network (RPN), and the above step S430 "determining the original detection loss of the first sample through the first branch network and the second branch network based on the instance-level label of the sample data in the first sample" can be implemented by the following steps:

[0148] Step S4301, determining a candidate box set and a region candidate loss of sample data in the first sample through the region candidate network;

[0149] Here, the region candidate network is used to generate all candidate regions in the first sample. The input of the region candidate network is the feature map of the first sample output by the backbone network, and the output of the region candidate network is the candidate box set of the first sample. The candidate box set includes two major categories of candidate boxes, one category of candidate boxes is a positive candidate box including the target, and the other category is a negative candidate box including the background.

[0150] The region candidate loss consists of two parts, classification loss and bounding box regression loss. The classification result of RPN has only two categories, namely background or target, which can be implemented using multi-category cross entropy loss or binary cross entropy loss; the bounding box regression loss uses smooth L1 loss.

[0151] Step S4302, determining the classification loss of the first sample through the second branch network based on the instance-level label of the sample data in the first sample and the candidate box set;

[0152] Here, the classification loss of the first sample through the second branch network can be calculated by a cross entropy function. First, based on the instance-level label of the sample data in the first sample, the category labels of the positive candidate boxes and the negative candidate boxes in the candidate box set are determined, and then combined with the detection result output by the second branch network, the classification loss of the first sample through the second branch network is determined.

[0153] Step S4303, determining the regression loss of the first sample through the first branch network based on the instance-level label of the sample data in the first sample and the positive candidate box containing the target in the candidate box set;

[0154] Here, the regression loss is only calculated on the positive candidate box, and the regression loss is used to adjust the prediction results of the boundary position of the candidate box.

[0155] In implementation, the position information of the positive candidate box in the first sample can be determined based on the instance-level label of the first sample, and then the regression loss of the first sample through the first branch network is calculated through a detection evaluation function (intersection-over-union) based on the position information of the positive candidate box in the first sample and the predicted position of the positive candidate box output by the first branch network.

[0156] Step S4303: Based on the region candidate loss, the classification loss and the regression loss, the first sample passes through the original detection loss of the first branch network and the second branch network.

[0157] Here, the original detection loss L of the first sample passing through the target detection model can be calculated by the following formula (3): f (x f ,q f ):

[0158] L f (x f ,q f )=L rpn (x f ,q f )+L cls (x f ,a p ,a n ,q f )+L loc (x f ,a p ,q f ) (3);

[0159] Among them, L rpn (x f ,q f ) is the first sample x f After the region proposal loss of the region proposal network, L cls (x f ,a p ,a n ,q f ) is the first sample x f Through the classification loss of the second branch network, L loc (x f ,a p ,q f ) is the first sample x f Through the regression loss of the first branch network, a p ,a nThe positive candidate box including the target and the negative candidate box including the background are output by the region proposal network respectively.

[0160] In this way, the region candidate loss generated by the first sample through the region candidate network, the classification loss generated by the second branch network, and the regression loss generated by the first branch network are calculated respectively, and the original detection loss of the final target detection model is determined, thereby improving the accuracy of target detection.

[0161] In some embodiments, determining the original detection loss of the first sample through the first branch network and the second branch network based on the instance-level label of the sample data in the first sample may also include: determining the target estimation loss of the sample data in the first sample through the first branch network based on the instance-level label of the sample data in the first sample and the candidate box set; determining the original detection loss of the first sample through the first branch network and the second branch network based on the region candidate loss, the classification loss, the target estimation loss and the regression loss.

[0162] Here, since the target estimation task is more related to positioning than classification, because it only measures the similarity between a candidate box and a general object without paying attention to the specific category, the target estimation task is added to the first branch network and trained in parallel with the classification task of the second branch network.

[0163] When the target estimation task is added to the first branch network, the original detection loss L of the first sample passing through the target detection model can be calculated by the following formula (4): f (x f ,q f ):

[0164] L f (x f ,q f )=L rpn (x f ,q f )+L cls (x f ,a p ,a n ,q f )+L obj (x f ,a p ,a n ,q f )+L loc (x f ,a p ,q f ) (4);

[0165] In this way, by introducing an additional target estimation task, the imbalance problem between foreground and background can be alleviated, and the average accuracy of model detection can be increased. At the same time, adding this task to the positioning branch can achieve higher performance improvement.

[0166] Figure 5A A schematic diagram of a target detection method provided in an embodiment of the present application is shown in FIG. Figure 5A As shown, the target detection method includes at least the following steps:

[0167] Step S510, obtaining a target image to be detected;

[0168] Here, the method of acquiring the target image may include: acquiring the target image using a camera of the image device itself; receiving an image from an external device; or reading the target image from a local database or a local memory.

[0169] Step S520, identifying the target image through the first branch network of the trained target detection model to obtain a detection frame of the target in the target image and a corresponding objective evaluation score;

[0170] Here, the target detection model is trained according to the model training method provided in any of the above embodiments. The target detection model can be, for example, a network structure such as CNN, R-CNN, Fast R-CNN, Faster R-CNN, etc. The embodiment of the present application does not limit the specific network structure of the target detection model.

[0171] The first branch network of the target detection model is used to perform positioning tasks. By inputting the target image into the first branch network of the target detection model, the location area of ​​the target in the target image, namely the detection box, can be determined. At the same time, the corresponding objective evaluation score can also be output, aiming to accurately locate and merge the foreground targets that have completed the classification to avoid multiple detections.

[0172] Step S530, identifying the target image through the second branch network of the target detection model to obtain the category of the target in the target image;

[0173] Here, the second branch network of the target detection model is used to perform the classification task. The target image is input into the second branch network of the target detection model, and the classification scores of the target in each category can be output to determine the category of the target.

[0174] In some implementations, the first branch network and the second branch network include the same two-layer fully connected layer structure but do not share network parameters.

[0175] Step S540: determining a target detection result of the target image based on the detection frame of the target, the objective evaluation score and the category of the target.

[0176] Here, the objective evaluation score output by the first branch network is multiplied by the classification score output by the second branch network to obtain the final target estimation probability, thereby determining the target detection result of the target image.

[0177] In an embodiment of the present application, first, a target image to be detected is obtained; then, the target image is recognized by a first branch network of a trained target detection model to obtain a detection frame of the target in the target image and a corresponding objective evaluation score; then, the target image is recognized by a second branch network of the target detection model to obtain a category of the target in the target image; finally, based on the detection frame of the target, the objective evaluation score and the category of the target, the target detection result of the target image is determined; in this way, by decoupling the positioning task and the classification task through the separated first branch network and the second branch network, the classification and recognition ability of the model for sparse long-tail categories can be improved, thereby improving the accuracy of target detection.

[0178] Figure 5B The following is a logical flow chart of the target detection model training method provided in the embodiment of the present application, such as Figure 5B As shown in the figure, the solid arrow flow is the processing flow of inputting the first sample 51 into the target detection network, and the dotted arrow flow is the processing flow of inputting the second sample 52 into the target detection network: the first sample 51 is sequentially extracted with features through the backbone network 53, and then the candidate box set 541 detected in the first sample 51 is output through the RPN network 54, that is, the region of interest (ROI) in the image, and the ROI feature 551 corresponding to the candidate box set 541 in the first sample 51 and the ROI feature 551 corresponding to the target box 521 in the second sample 52 are determined in the ROI-Align 55 module; the ROI feature 551 extracted from the first sample 51 is respectively input into the first branch network composed of the fully connected layer 56 and the fully connected layer 57, and the second branch network composed of the fully connected layer 58 and the fully connected layer 59, and the ROI feature 551 extracted from the second sample 52 is input into the second branch network composed of the fully connected layer 58 and the fully connected layer 59. The regression loss and target estimation loss are calculated through the output of the first branch network, and the classification loss is calculated through the output of the second branch network.

[0179] Figure 5C The system architecture diagram of the target detection model training method provided in the embodiment of the present application is as follows: Figure 5CAs shown, feature extraction is performed on the first sample 51 and a candidate box set is generated through the RPN network 54, and feature extraction is performed directly on the second sample with a pseudo label; ROI-Align is used to determine the ROI features corresponding to the candidate boxes in the first sample 51, and the positioning loss and classification loss are determined respectively; at the same time, ROI-Align 55 is used to determine the ROI features corresponding to the target box in the second sample 52, and the classification loss is further determined.

[0180] It can be seen that the second branch network for the classification task processes both the second sample, i.e., the weakly labeled data, and the first sample, i.e., the fully labeled data, while the first branch network for the localization task only processes the first sample, i.e., the fully labeled data. In other words, the second sample is jointly trained with the first sample in a classification enhancement manner.

[0181] Unlike the target detection method in the related art that only relies on the fully labeled first sample, the embodiment of the present application uses the instance-level label of the first sample to obtain the second sample with image-level labels that naturally exist on the network or in the community, and trains with the second sample to improve the accuracy performance of long-tail target detection, which is less prone to overfitting. At the same time, the embodiment of the present application uses the information of the image-level label to generate accurate pseudo-labels for the second sample in difficult long-tail situations, overcome the influence of noise, and greatly improve the performance of the model. Finally, the classification enhancement method is used to efficiently utilize the additionally obtained second samples, while improving performance without introducing too much computational burden.

[0182] Based on the foregoing embodiments, the embodiments of the present application further provide a model training device, which includes the modules included, and the sub-modules and units included in each module, which can be implemented by a processor in an electronic device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0183] Figure 6 A schematic diagram of the structure of a model training device provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the apparatus 600 includes a first acquisition module 610, a first determination module 620, a second determination module 630, a third determination module 640 and a training module 650, wherein:

[0184] The first acquisition module 610 is used to acquire a first sample with an instance-level label and a second sample with an image-level label; the second sample is acquired based on the instance-level label of the first sample; the first determination module 620 is used to determine the pseudo-label of the sample data in the second sample through a pre-trained target detection model; the second determination module 630 is used to determine the original detection loss of the target detection model based on the instance-level label of the sample data in the first sample; the third determination module 640 is used to determine the classification enhancement loss of the target detection model based on the pseudo-label of the sample data in the second sample; the training module 650 is used to train the target detection model using the first sample and the second sample based on the original detection loss and the classification enhancement loss.

[0185] In some possible embodiments, the first acquisition module 610 includes: a first acquisition submodule, used to acquire the first sample with an instance-level label from a sample training set; a first determination submodule, used to determine at least one category included in the sample data in the first sample based on the instance-level label; and a second acquisition submodule, used to acquire a second sample corresponding to the corresponding category based on the at least one category, to obtain the second sample with an image-level label.

[0186] In some possible embodiments, the at least one category is the category to which all sample data in the first sample belong, and the all sample data include normal data and long-tail data; the second acquisition submodule is also used to acquire the second sample corresponding to the category through a preset image data set to obtain the second sample with an image-level label.

[0187] In some possible embodiments, the first determination module 620 includes: a second determination submodule, used to input the second sample into the pre-trained target detection model to obtain a set of candidate boxes for sample data in the second sample; a third determination submodule, used to determine the classification confidence of the candidate boxes in the candidate box set under the image-level labels of the corresponding sample data; a sorting submodule, used to sort the classification confidence of each of the candidate boxes in the candidate box set to obtain the target box with the highest classification confidence score; and a fourth determination submodule, used to generate a pseudo label for the sample data in the second sample based on the target box with the highest classification confidence score.

[0188] In some possible embodiments, the target detection model includes a first branch network for performing a positioning task and a second branch network for performing a classification task; the second determination module 630 is also used to determine the original detection loss of the first sample through the first branch network and the second branch network based on the instance-level label of the sample data in the first sample; the third determination module 640 is also used to determine the classification enhancement loss of the second sample through the second branch network of the target detection model based on the pseudo-label of the sample data in the second sample.

[0189] In some possible embodiments, the training module 640 includes: a fifth determination submodule, used to determine the task balance coefficient of the classification task and the positioning task; a combination submodule, used to linearly sum the original detection loss and the classification enhancement loss based on the task balance coefficient to obtain the optimized target loss of the target detection model; a training submodule, used to train the target detection model using the first sample and the second sample under the supervision of the optimized target loss.

[0190] In some possible embodiments, the target detection model also includes a region candidate network, and the second determination module 630 includes: a candidate box loss determination submodule, which is used to determine the candidate box set and the region candidate loss of the sample data in the first sample through the region candidate network; a classification loss determination submodule, which is used to determine the classification loss of the sample data in the first sample through the second branch network based on the instance-level label of the sample data in the first sample and the candidate box set; a regression loss determination submodule, which is used to determine the regression loss of the sample data in the first sample through the first branch network based on the instance-level label of the sample data in the first sample and the positive candidate box containing the target in the candidate box set; a sixth determination submodule, which is used to determine the original detection loss of the target detection model based on the region candidate loss, the classification loss and the regression loss.

[0191] In some possible embodiments, the second determination module 630 also includes an estimation loss determination submodule, which is used to determine the target estimation loss of the sample data in the first sample through the first branch network based on the instance-level label of the sample data in the first sample and the set of candidate boxes; accordingly, the sixth determination submodule is also used to determine the original detection loss of the first sample through the first branch network and the second branch network based on the region candidate loss, the classification loss, the target estimation loss and the regression loss.

[0192] In some possible embodiments, the pseudo label of the second sample also includes position information of a target box in the second sample, and the target box is a candidate box corresponding to the highest classification confidence in the second sample; the training module also includes: a target area determination submodule, which is used to determine the target area of ​​the sample data in the corresponding second sample based on the position information of the target box in the sample data in the second sample before the next iteration training; a fusion submodule, which is used to fuse the target area of ​​the sample data in at least one of the second samples to obtain a synthetic sample image; the training module is also used to train the target detection model based on the original detection loss and the classification enhancement loss using the first sample, the second sample and the synthetic sample image.

[0193] In some possible embodiments, the fusion submodule includes: a data enhancement unit, used to perform data enhancement processing on a target area of ​​sample data in the second sample; the data enhancement processing includes at least one of the following operations: random scaling, cropping or mosaic; a fusion unit, used to fuse the target area after at least one enhancement processing to obtain a synthetic sample image.

[0194] In some possible embodiments, the first branch network and the second branch network include the same two-layer fully connected layer structure, and / or the first branch network and the second branch network are trained in parallel.

[0195] It should be noted that the description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0196] Based on the foregoing embodiments, the embodiments of the present application further provide a model training device, which includes the modules included, and the sub-modules and units included in each module, which can be implemented by a processor in an electronic device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit, a microprocessor, a digital signal processor or a field programmable gate array, etc.

[0197] Figure 7 A schematic diagram of the structure of a target detection device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the device 700 includes a second acquisition module 710, a first identification module 720, a second identification module 730 and a fourth determination module 740, wherein:

[0198] The second acquisition module 710 is used to acquire a target image to be detected; the first recognition module 720 is used to recognize the target image through the first branch network of the trained target detection model to obtain a detection frame of the target in the target image and a corresponding objective evaluation score; the second recognition module 730 is used to recognize the target image through the second branch network of the target detection model to obtain the category of the target in the target image; the fourth determination module 740 is used to determine the target detection result of the target image based on the detection frame of the target, the objective evaluation score and the category of the target.

[0199] It should be noted that the description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0200] It should be noted that in the embodiment of the present application, if the above-mentioned model training method or target detection method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable an electronic device (which can be a smart phone with a camera, a tablet computer, etc.) to execute all or part of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any preset hardware and software combination.

[0201] Correspondingly, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the target detection method or model training method described in any of the above embodiments are implemented. Correspondingly, in an embodiment of the present application, a chip is also provided, and the chip includes a programmable logic circuit and / or program instructions, and when the chip is running, it is used to implement the steps of the target detection method or model training method described in any of the above embodiments. Correspondingly, in an embodiment of the present application, a computer program product is also provided, and when the computer program product is executed by a processor of an electronic device, it is used to implement the steps of the target detection method or model training method described in any of the above embodiments.

[0202] Based on the same technical concept, an embodiment of the present application provides an electronic device for implementing the target detection method or model training method recorded in the above method embodiment. Figure 8 A hardware entity diagram of an electronic device provided in an embodiment of the present application, such as Figure 8 As shown, the electronic device 800 includes a memory 810 and a processor 820, and the memory 810 stores a computer program that can be executed on the processor 820. When the processor 820 executes the program, the steps in any target detection method or model training method described in the embodiments of the present application are implemented.

[0203] The memory 810 is configured to store instructions and applications executable by the processor 820, and can also cache data to be processed or processed by the processor 820 and various modules in the electronic device (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0204] When the processor 820 executes the program, the steps of any of the above-mentioned target detection methods or model training methods are implemented. The processor 820 generally controls the overall operation of the electronic device 800.

[0205] The processor may be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor. It is understandable that the electronic device that implements the functions of the processor may also be other, and the embodiments of the present application are not specifically limited.

[0206] The above-mentioned computer storage medium / memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM) and the like; it can also be various electronic devices including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0207] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0208] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these preset features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments.

[0209] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0210] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms. The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment scheme of the present application.

[0211] In addition, all functional units in the embodiments of the present application can be integrated into one processing unit, or each unit can be used as a unit separately, or two or more units can be integrated into one unit; the above-mentioned integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional units. Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium, including several instructions for the automatic test line of the device to execute all or part of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks or optical disks. The methods disclosed in the several method embodiments provided in the present application can be arbitrarily combined without conflict to obtain new method embodiments. The features disclosed in the several method or device embodiments provided in the present application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments. The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A model training method, characterized in that: The method comprises: Obtain a first sample with an instance-level label and a second sample with an image-level label; the second sample is obtained based on the instance-level label of the first sample; Determining a pseudo label of the sample data in the second sample by using a pre-trained target detection model; Determining an original detection loss of the object detection model based on instance-level labels of sample data in the first sample; Determining a classification enhancement loss of the object detection model based on the pseudo labels of the sample data in the second sample; Based on the original detection loss and the classification enhancement loss, training the object detection model using the first sample and the second sample; The target detection model includes a first branch network for performing a positioning task and a second branch network for performing a classification task; The determining, based on the instance-level label of the sample data in the first sample, the original detection loss of the target detection model comprises: Based on the instance-level label of the sample data in the first sample, determining the original detection loss of the first sample through the first branch network and the second branch network; The determining, based on the pseudo-label of the sample data in the second sample, the classification enhancement loss of the target detection model comprises: Based on the pseudo-label of the sample data in the second sample, a classification enhancement loss of the second sample passing through the second branch network of the target detection model is determined.

2. The method according to claim 1, characterized in that The obtaining of a first sample with an instance-level label and a second sample with an image-level label comprises: Obtain the first sample with instance-level label from the sample training set; Based on the instance-level label, determining at least one category included in the first sample; Based on the at least one category, a second sample corresponding to the category is obtained to obtain the second sample with an image-level label.

3. The method according to claim 2, characterized in that The at least one category is the category to which all sample data in the first sample belong, and the all sample data include normal data and long-tail data; The acquiring, based on the at least one category, a second sample corresponding to the category to obtain the second sample with an image-level label includes: The second sample corresponding to each of the categories is obtained through a preset image data set to obtain the second sample with an image-level label.

4. The method according to any one of claims 1 to 3, characterized in that The determining, by using the pre-trained target detection model, a pseudo label of the sample data in the second sample comprises: Inputting the second sample into the pre-trained object detection model to obtain a set of candidate boxes of sample data in the second sample; Determine the classification confidence of each candidate box in the candidate box set under the image-level label of the corresponding sample data; Sorting the classification confidences of the candidate frames in the candidate frame set to obtain the target frame with the highest classification confidence score; Based on the target box with the highest classification confidence score, a pseudo label of the sample data in the second sample is generated.

5. The method according to claim 1, characterized in that The method of training the target detection model by using the first sample and the second sample based on the original detection loss and the classification enhancement loss includes: Determining a task balance coefficient between the classification task and the positioning task; Based on the task balance coefficient, linearly summing the original detection loss and the classification enhancement loss to obtain an optimized target loss of the target detection model; The target detection model is trained using the first sample and the second sample under the supervision of the optimized target loss.

6. The method according to claim 1, characterized in that The target detection model further includes a region candidate network, and determining the original detection loss of the first sample through the first branch network and the second branch network based on the instance-level label of the sample data in the first sample includes: Determining, by means of the region candidate network, a candidate box set and a region candidate loss of sample data in the first sample; Determine, based on the instance-level label of the sample data in the first sample and the set of candidate boxes, a classification loss of the first sample through the second branch network; Determine, based on the instance-level label of the sample data in the first sample and the positive candidate box containing the target in the candidate box set, the regression loss of the first sample through the first branch network; Based on the region candidate loss, the classification loss, and the regression loss, an original detection loss of the first sample through the first branch network and the second branch network is determined.

7. The method according to claim 6, characterized in that The determining, based on the instance-level label of the sample data in the first sample, the original detection loss of the first sample through the first branch network and the second branch network further includes: Determine, based on the instance-level label of the sample data in the first sample and the set of candidate boxes, a target estimation loss of the first sample passing through the first branch network; Based on the region candidate loss, the classification loss, the target estimation loss, and the regression loss, an original detection loss of the first sample through the first branch network and the second branch network is determined.

8. The method according to any one of claims 1 to 3, characterized in that The pseudo label of the second sample further includes position information of a target box in the second sample, where the target box is a candidate box corresponding to the highest classification confidence in the second sample; The method of training the target detection model using the first sample and the second sample based on the original detection loss and the classification enhancement loss further includes: Before each iteration of training, determining a target area corresponding to the sample data in the second sample based on position information of the target box in the sample data in the second sample; fusing a target region of sample data in at least one of the second samples to obtain a synthesized sample image; Based on the original detection loss and the classification enhancement loss, the target detection model is trained using the first sample, the second sample and the synthesized sample image.

9. The method according to claim 8, characterized in that The step of fusing a target area of ​​sample data in at least one of the second samples to obtain a synthesized sample image includes: Performing data enhancement processing on the target area of ​​the sample data in the second sample; the data enhancement processing includes at least one of the following operations: random scaling, cropping or mosaic; The at least one enhanced target region is fused to obtain a synthetic sample image.

10. The method according to any one of claims 1 to 3, characterized in that: The first branch network and the second branch network include the same two-layer fully connected layer structure, and / or the first branch network and the second branch network are trained in parallel.

11. A target detection method, characterized in that: The method comprises: Acquire the target image to be detected; The target image is recognized by a first branch network of a trained target detection model to obtain a detection frame of the target in the target image and a corresponding objective evaluation score; the trained target detection model is a target detection model trained by the model training method according to any one of claims 1 to 10; Recognizing the target image through the second branch network of the target detection model to obtain the category of the target in the target image; An object detection result of the object image is determined based on the detection frame of the object, the objective evaluation score, and the category of the object.

12. A model training device, characterized in that: The device comprises a first acquisition module, a first determination module, a second determination module, a third determination module and a training module, wherein: The first acquisition module is used to acquire a first sample with an instance-level label and a second sample with an image-level label; the second sample is acquired based on the instance-level label of the first sample; The first determination module is used to determine the pseudo labels of the sample data in the second sample by using a pre-trained target detection model; The second determination module is used to determine the original detection loss of the target detection model based on the instance-level label of the sample data in the first sample; The third determination module is used to determine the classification enhancement loss of the object detection model based on the pseudo labels of the sample data in the second sample; The training module is used to train the target detection model using the first sample and the second sample based on the original detection loss and the classification enhancement loss; Wherein, the target detection model includes a first branch network for performing a positioning task and a second branch network for performing a classification task; The second determination module is further used to determine the original detection loss of the first sample through the first branch network and the second branch network based on the instance-level label of the sample data in the first sample; The third determination module is further used to determine the classification enhancement loss of the second sample through the second branch network of the target detection model based on the pseudo label of the sample data in the second sample.

13. A target detection device, characterized in that: The device comprises a second acquisition module, a first identification module, a second identification module and a fourth determination module, wherein: The second acquisition module is used to acquire the target image to be detected; The first recognition module is used to recognize the target image through a first branch network of a trained target detection model to obtain a detection frame of the target in the target image and a corresponding objective evaluation score; the trained target detection model is a target detection model trained by the model training method according to any one of claims 1 to 10; The second recognition module is used to recognize the target image through the second branch network of the target detection model to obtain the category of the target in the target image; The fourth determination module is used to determine the target detection result of the target image based on the detection frame of the target, the objective evaluation score and the category of the target.

14. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the processor implements the steps of the method according to any one of claims 1 to 10, or implements the steps of the method according to claim 11.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 10, or implements the steps of the method according to claim 11.

Images