Multi-class object detection method and model training method and device thereof

Abstract

The application relates to the technical field of target detection, and provides a multi-class target detection method and a model training method and device thereof. The method obtains different teacher networks through pre-training, and detects corresponding class targets in training pictures containing multiple target classes and only having partial target class labels by using each teacher network, so as to obtain reference feature maps in different detection stages. Then, the loss values of corresponding prediction feature maps output by a student network to be trained are calculated by using the reference feature maps obtained by all the teacher networks, and finally, the loss values are used for back propagation training, so that a multi-class target detection model is obtained. The method can use an incomplete labeled data set to train the multi-class target detection model, and can improve training effect and efficiency.

Description

Technical Field

[0001] This application relates to the field of target detection technology, and in particular to a multi-category target detection method and its model training method and apparatus. Background Technology

[0002] When a project's object detection model needs to simultaneously detect and recognize multiple objects—for example, detecting all fruits in an image and identifying their species—it typically requires first organizing the relevant sample dataset before training the object detection model. During dataset collection and training, it's generally desirable to collect a complete dataset, meaning that every image in the dataset contains annotations for all objects to be detected.

[0003] However, in real-world projects, a common scenario is this: there's a dataset A with labeled objects A; and a dataset B with labeled objects B. But in dataset A, even if object B appears in an image, it's not labeled, and the same applies to dataset B. Therefore, directly merging these two datasets A and B to train a multi-object object detection model is not feasible. Summary of the Invention

[0004] In view of this, embodiments of this application provide a multi-category target detection method and its model training method and apparatus. In a first aspect, embodiments of this application provide a method for inputting training images into different pre-trained teacher networks for target detection, and outputting reference feature maps for different detection stages in each teacher network; wherein, different teacher networks are used to detect different categories of targets; the training images contain multiple target categories and have partial target category annotations; Based on the reference feature maps at different detection stages, positive and negative sample data of the training image in the corresponding category of object detection are obtained; The training images are input into the student network for multi-class object detection, and the predicted feature maps are output at each detection stage. Based on the positive and negative sample data, the reference feature map, and the predicted feature map, calculate the loss value when performing multi-class target detection through the student network; The loss value is used for backpropagation to continue training the student network until a preset training stopping condition is met, thus obtaining a multi-class object detection model.

[0005] In some embodiments, outputting reference feature maps for different detection stages in each teacher network includes: outputting reference feature maps obtained by each teacher network at different detection stages according to a preset size, wherein each point in each reference feature map contains a preset number of anchor boxes, and the anchor boxes are used to obtain the positive and negative sample data through parsing.

[0006] In some embodiments, obtaining positive and negative sample data of the training image in the corresponding category of object detection based on the reference feature maps at different detection stages includes: During the target detection process of the corresponding category, the anchor boxes at the corresponding positions in the reference feature map in each detection stage are assigned as positive or negative samples according to the preset allocation rules. The distinguished positive and negative samples are described in matrix form to obtain the positive and negative sample data of the training image in the corresponding category of object detection.

[0007] In some embodiments, the knowledge distillation loss value between the reference feature map and the predicted feature map in each detection stage is calculated using the positive and negative sample data of each detection stage. The total knowledge distillation loss value is obtained by weighting the knowledge distillation loss values ​​from all detection stages. Calculate the target detection loss value between the predicted feature map and the ground truth feature map obtained based on the target category annotations contained in the training images; The loss value for multi-class target detection is calculated by using the total loss value of knowledge distillation and the target detection loss value.

[0008] In some embodiments, the different detection stages include three stages: a regression detection stage, a target determination stage, and a target category determination stage. In the target category determination stage, each teacher network outputs a reference feature map for determining only one target category; the student network outputs corresponding predicted feature maps for determining all target categories in the target category determination stage.

[0009] In some embodiments, the number of teacher networks does not exceed the total number of target categories to be detected; the different teacher networks are obtained through pre-training, including: Each sample image containing multiple categories of targets is divided into several datasets with different target category labels according to different partial target annotations; Using different datasets, several constructed neural networks are trained for different target detection tasks to obtain several teacher networks for detecting different types of targets, with the same number of datasets as the datasets.

[0010] Secondly, embodiments of this application also provide a multi-category target detection method, including: The target image is input into the multi-class target detection model obtained by the multi-class target detection model training method to perform target detection, and the prediction results of all categories of targets present in the target image are obtained.

[0011] Thirdly, embodiments of this application also provide a multi-category object detection model training device, comprising: The teacher network inference module is used to input training images into different pre-trained teacher networks for object detection and output reference feature maps for different detection stages of each teacher network; wherein, different teacher networks are used to detect different categories of targets; the training images contain multiple target categories and have partial target category annotations; The positive and negative sample acquisition module is used to obtain positive and negative sample data of the training image in the corresponding category of target detection based on the reference feature map at different detection stages. The student network training module is used to input the training images into the student network for multi-class object detection and output the predicted feature maps at each detection stage. The loss calculation module is used to calculate the loss value when performing multi-class target detection through the student network based on the positive and negative sample data, the reference feature map, and the predicted feature map. The student network training module is also used to perform backpropagation using the loss value to continue training the student network until a preset training stopping condition is met, thereby obtaining a multi-class object detection model.

[0012] Fourthly, embodiments of this application also provide a terminal device, the terminal device including a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the multi-class target detection model training method or the multi-class target detection method.

[0013] Fifthly, embodiments of this application also provide a readable storage medium storing a computer program, which, when executed on a processor, implements the multi-class target detection model training method or the multi-class target detection method.

[0014] The embodiments of this application have the following beneficial effects: The multi-class object detection model training method of this application pre-trains different teacher networks and uses each teacher network to perform corresponding class object detection on training images with only partial object class annotations to obtain corresponding reference feature maps at different detection stages. Then, these obtained reference feature maps are used to calculate the loss value of the corresponding predicted feature map output by the student network to be trained. Finally, backpropagation training is performed using the loss value to obtain a multi-class object detection model. This training method not only eliminates the need for manual annotation, supplementation, and adjustment of the existing dataset, but also fully utilizes all dataset information for model training through knowledge distillation, achieving the goal of simultaneously improving training efficiency and effectiveness. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A schematic diagram of a typical multi-target detection network including a binary classifier is shown. Figure 2 This paper shows a first flowchart of a multi-category object detection model training method according to an embodiment of the present application; Figure 3 A second flowchart of the multi-class object detection model training method according to an embodiment of this application is shown; Figure 4 A schematic diagram of two-class target detection using the training method of this application is shown; Figure 5 The third flowchart of the multi-class object detection model training method according to an embodiment of this application is shown; Figure 6 A flowchart of a multi-category target detection method according to an embodiment of this application is shown; Figure 7 A schematic diagram of the structure of the multi-class target detection model training method according to an embodiment of this application is shown. Detailed Implementation

[0017] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0018] The components of the embodiments of this application described and illustrated in the accompanying drawings can be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of this application provided in the drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0019] In the following text, the terms "comprising," "having," and their cognates, which may be used in various embodiments of this application, are intended only to indicate a particular feature, number, step, operation, element, component, or combination thereof, and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations thereof, or adding the possibility of one or more combinations thereof. Furthermore, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0020] Unless otherwise specified, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of this application pertain. Terms (such as those defined in commonly used dictionaries) shall be interpreted as having the same meaning as in their contextual meaning in the relevant technical field and shall not be construed as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of this application.

[0021] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0022] Typically, given two datasets A and B with only partial object annotations, the simplest way to detect both types of objects is to train a detection model for object A using dataset A and a detection model for object B using dataset B. However, in real-world projects, due to limited resources such as terminal devices, usually only one detection model can be used to detect both object A and object B simultaneously. Since dataset A may contain a large number of unlabeled objects B, and similarly, dataset B may contain a large number of unlabeled objects A, the following are some common approaches to achieve simultaneous detection of multiple objects: The first approach is to manually add annotations for object B in dataset A and add annotations for object A in dataset B, and then train a detection model. As you can imagine, adding annotations to these datasets will require a significant amount of annotation costs.

[0023] The second approach involves training two object detection networks separately on datasets A and B. These networks are then used to infer pseudo-labels for missing categories on both datasets. Finally, the network is trained on the completed dataset. This approach is highly sensitive to the quality of the pseudo-labels and still requires significant time to fine-tune them.

[0024] The third option is, according to... Figure 1 The network structure shown is used to design an object detection network model. The Bockbone is responsible for feature extraction, the Reg Head in the Heads is responsible for calculating regression detection boxes, the ObjHead is responsible for determining whether an object is an object, and the Cls Head is responsible for determining the object's category. It's important to note that the Cls Head uses multiple binary classifiers (instead of the common softmax classifier). During training, when training on dataset A, only Cls Head A is trained, and Cls Head B is disabled; similarly, when training on dataset B, only Cls Head B is trained, and Cls Head A is disabled. While this scheme is basically usable, the information about object B appearing in dataset A is not effectively utilized.

[0025] To address the challenges of training multi-class object detection models using classic methods when multiple datasets exist and each dataset contains incomplete object category annotations, this application proposes a novel method for training multi-class object detection models based on the concept of knowledge distillation using multiple datasets with only partial category annotations. This method aims to improve the efficiency and effectiveness of model training.

[0026] The training method for this multi-class object detection model will be described in detail below with reference to some specific embodiments.

[0027] Exemplary, such as Figure 2 As shown, the training method for this multi-class object detection model includes: S110, the training images are input into different pre-trained teacher networks for object detection, and reference feature maps of different detection stages in each teacher network are output; wherein, different teacher networks are used to detect different categories of objects.

[0028] Based on the principle of knowledge distillation, the core idea is to first train a complex network model, and then use the output of this complex network and the real labels of the data to train a smaller network. The complex network model is called the Teacher network, and the smaller network model is called the Student network.

[0029] The training images contain multiple target categories and have partial target category annotations. In this application, multiple teacher networks are pre-trained using multiple datasets with partial target category annotations. These teacher networks are then used to perform corresponding target category detection on each input training image, thereby obtaining feature maps for each of the corresponding detection stages. It can be understood that since the feature maps output by each trained teacher network will be used as a reference when training the student network, they are referred to here as reference feature maps.

[0030] In one implementation, such as Figure 3 As shown, before step S110, the method further includes: S100: Each sample image containing multiple categories of targets is divided into several datasets with different target category labels according to different partial target labels; then, the constructed neural networks are trained for different target detection using different datasets to obtain several teacher networks for detecting different categories of targets, which are equal in number to the number of datasets.

[0031] The network layer structures of the various neural networks used as the teacher network can be the same, but the internal network parameters may differ due to the detection of different categories of targets.

[0032] It is worth noting that the number of teacher networks will not exceed the total number of target categories to be detected, and can depend on the size of the existing dataset. For example, each teacher network can be used to detect one target category, or one teacher network can be used to detect two or more target categories. Since the annotation information in the existing dataset is incomplete, it is not possible to directly detect all target categories.

[0033] For example, in order to train a program that can detect target O A O B and O C In a three-object model, assuming that there are a total of three object categories in each sample image, but each image only labels one or two object categories (i.e., the labeling information is incomplete), then all sample images with the same object category label can be grouped into one dataset, resulting in a dataset D with different object category labels. A D B and D C Furthermore, using dataset D A To train neural network S A To achieve target O A The detection utilizes dataset D B To train neural network S B For target O B The detection, and the use of dataset D CTo train neural network S C For target O C The detection process yields three teacher networks T, which can be used to detect three target categories A, B, and C respectively. A T B T C .

[0034] In another scenario, if each sample image is labeled with two object categories simultaneously, that is, if a dataset D1 contains two object categories (such as O...), then... A O B At the same time, it was also labeled, while another dataset, D2, labeled the target O. C After labeling, we can then use dataset D1 to train a dataset that can simultaneously detect target O. A O B The teacher network T1 is trained using dataset D2 to detect target O. C The teacher's network T2.

[0035] It is understandable that by setting up multiple teacher networks with an equal number of target categories to be detected, corresponding feature maps can be output for detecting the corresponding target categories. Since these teacher networks have been trained, the output feature maps meet the prediction accuracy requirements. Therefore, using them as a reference when training student networks allows student networks to learn the target recognition and detection capabilities of all teacher networks, while also achieving higher accuracy and lower resource consumption.

[0036] In the target detection process, multiple tasks are often required, such as regression and classification. In one implementation, the different detection stages described in this application mainly include three stages: a regression detection box calculation stage, a target determination stage, and a target category determination stage. The regression detection box calculation stage is mainly used to calculate the position information of the regression detection box that approximates the target; the target determination stage is mainly used to determine whether the grid containing the point belongs to the target (foreground) or the background; and the target category determination stage is mainly used to determine what target category the target belongs to.

[0037] It's understandable that a teacher network can only output reference feature maps for some target category determinations during the target category determination phase. In contrast, a student network can output corresponding predicted feature maps for all target categories during the target category determination phase; that is, it will output predicted feature maps for several target categories if there are several target categories.

[0038] Furthermore, when outputting reference feature maps, the reference feature maps obtained by each teacher network at different detection stages can be output according to a preset size. It is worth noting that each point (grid) in each output reference feature map contains a preset number of anchor boxes.

[0039] like Figure 4 As shown, taking two types of object detection (object A and object B) as an example, when a training image is input into the teacher network A to detect object A, the corresponding feature maps obtained by the teacher network A in three stages can be output, denoted as follows: , , Similarly, when the training image is input into teacher network B for target B detection, the corresponding feature maps obtained by teacher network A in the three stages can be output, denoted as follows: , , The size of these reference feature maps is Furthermore, each point in the graph contains K anchors.

[0040] It is understandable that by further analyzing the anchors in these reference feature maps, the distribution of positive and negative samples when the training image is used to detect the corresponding target category can be obtained, so as to calculate the loss of the target category, foreground and background distinction, etc. in the corresponding detection stage.

[0041] S120: Based on the reference feature maps of different detection stages, obtain the positive and negative sample data of the training image in the corresponding category object detection process.

[0042] For example, during the corresponding category object detection process, anchor boxes at corresponding positions in the reference feature maps of each detection stage can be assigned as positive or negative samples according to a preset allocation rule; then, the distinguished positive and negative samples are described in matrix form to obtain the positive and negative sample data of the training image in the corresponding category object detection.

[0043] There are various ways to assign anchor boxes in these reference feature maps to positive and negative samples according to the preset assignment rules, which are not limited here. For example, in one implementation, for each anchor, the maximum IoU of the ground truth (GT) that best matches the anchor can be calculated, and thresholds for positive and negative samples can be set respectively to achieve the matching of positive and negative samples. Specifically, if the maximum IoU is greater than the set positive sample threshold, the anchor box is considered a positive sample; if the maximum IoU is less than the set negative sample threshold, the anchor is considered a negative sample. Here, IoU is used to measure the correlation between the true value and the predicted value, and is the result of dividing the overlapping part of two regions by the set of the two regions.

[0044] For example, in another implementation, the IoU between each anchor and its corresponding ground plane can be calculated, and the sum of the mean and standard deviation can be used as the positive sample screening threshold. Anchors with an IoU greater than the screening threshold with the ground plane can then be identified, and anchors whose center is inside the ground plane and whose IoU with the ground plane is greater than the screening threshold can be used as positive samples.

[0045] Alternatively, in another implementation, positive and negative samples can be distinguished using a probability distribution. For example, a probability distribution can be obtained by calculating the score of the anchor at the corresponding position in the reference feature map during the corresponding detection phase; this score reflects the quality of the bounding box classification and localization. Then, this probability distribution is used to distinguish whether the anchor is a positive or negative sample. It can be understood that by scoring each anchor, an evaluation can be obtained as to whether the model considers the anchor to be very useful for detecting the target object, thereby determining whether the anchor belongs to a positive or negative sample.

[0046] Therefore, these anchors can be assigned positive and negative samples. To facilitate subsequent loss calculation, the distinguished positive and negative samples can be described in matrix form. For example, anchors that are positive samples can be represented by "1" and anchors that are negative samples can be represented by "0", so as to obtain the positive and negative sample data of the training image in the corresponding category object detection process.

[0047] S130: Input the training image into the student network for multi-class object detection, and output the predicted feature maps at each detection stage.

[0048] like Figure 4 As shown, unlike the teacher network which only contains some classification decision branches, the student network in this application contains all classification decision branches for different target categories.

[0049] As an example, inputting the training images described above into the student network yields feature maps for the corresponding detection stages. For instance, using the three stages mentioned above, the corresponding feature maps can be predicted and output. , , and Then, the training loss is calculated based on these predicted feature maps.

[0050] S140, Based on the positive and negative sample data, the reference feature map, and the predicted feature map, calculate the loss value of the student network when performing multi-class target detection.

[0051] In one implementation, such as Figure 5 As shown, step S140 above includes the following sub-steps: S141, using the positive and negative sample data, calculate the knowledge distillation loss value between the reference feature map and the predicted feature map for each detection stage.

[0052] S142, calculate the total knowledge distillation loss value by weighting the knowledge distillation loss values ​​of all detection stages.

[0053] For example, taking the detection of the two target categories and the three detection stages mentioned above as an example, the expression for calculating the knowledge distillation loss value is as follows: ; ; ; In the formula, , , These represent the knowledge distillation loss values ​​for the regression detection box calculation stage, the target determination stage, and the target category determination stage, respectively. , This represents the positive and negative sample matrices obtained by the inference of each of the two teachers' networks; This represents the reference feature map for target A during the regression detection box calculation phase. With predicted feature map The loss value between This represents the reference feature map for target B during the regression detection box calculation phase. With predicted feature map The loss value between; where ij represents the value of the j-th anchor at the i-th point, 0 represents that the point is a negative sample, and 1 represents that the point is a positive sample; Similarly, This represents the reference feature map for target A during the target determination phase. With predicted feature map The loss value between This represents the reference feature map for target B during the target determination phase. With predicted feature map The loss value between; This represents the reference feature map for target A during the target category determination stage. With predicted feature map The loss value between This represents the reference feature map for target B during the target category determination stage. With predicted feature map The loss value between.

[0054] It is understandable that the calculation of the knowledge distillation loss value is related to the number of target categories to be detected and the detection stage. Specifically, the calculation of the loss value at a particular detection stage... When calculating the loss function, commonly used loss functions such as L1, L2, Smooth L1, and Smooth L2 can be used, and no specific limitation is made here.

[0055] Furthermore, based on preset weights, the total loss value of this knowledge distillation is... It can be calculated using the following formula: ; In the formula, , and Each of the three stages has its own preset weight.

[0056] S143, calculate the target detection loss value between the predicted feature map and the real feature map obtained based on the target category annotations contained in the training image. In addition to considering the knowledge distillation loss, since the dataset to which the training image belongs is known, the target detection loss for predicting the target class can be calculated using the partial target class labeling information, which may include classification loss and regression loss.

[0057] Taking the two types of object detection mentioned above as examples, if the training image belongs to dataset A, then based on the ground truth map corresponding to the annotation of object A, the regression loss between the predicted feature map and the ground truth map of object A in the regression detection box calculation stage can be calculated. L reg For example, it can be calculated using loss functions such as L1, L2, Smooth L1, and Smooth L2. Simultaneously, during the target category determination stage, the category loss for target A can be calculated. L cls For example, the cross-entropy loss function can be used to calculate it. Finally, the classification loss and regression loss are added together or weighted to obtain the object detection loss value L.

[0058] S144, using the total loss value of knowledge distillation and the target detection loss value, calculate the loss value when performing multi-class target detection through the student network.

[0059] Therefore, the total loss value can be obtained by summing all types of loss values. For example, it can be represented as follows: ; In the formula, This represents the total loss value; This represents the preset weight value, which can be set according to requirements.

[0060] S150, backpropagation is performed using the loss value to continue training the student network until the preset training stopping condition is met, thus obtaining a multi-class object detection model.

[0061] During training, gradient backpropagation is performed based on the calculated total loss value. Through backpropagation, for example, gradient descent can be used to iterate the weights in the student network and continue training until preset conditions are met, resulting in a trained student network that can be used as a multi-class object detection model. The principle of backpropagation in neural networks is not the focus of this application; please refer to published literature for details, which will not be elaborated here.

[0062] The aforementioned preset training stopping conditions are used to determine when to stop training. For example, they may include, but are not limited to, the total loss value being less than a preset threshold (i.e., the total loss value being small enough, such as approaching 0 or being within a certain range), reaching the iteration number threshold, or the total loss value and the iteration number simultaneously meeting the requirements, etc. No specific limitations are made here.

[0063] This embodiment's multi-class object detection model training method utilizes the concept of knowledge distillation. It pre-trains multiple teacher networks using a dataset with partial object class annotations. Then, it uses the reference feature maps of each teacher network at different detection stages as a reference for training the student network. The training loss is calculated using both positive and negative sample data. Finally, backpropagation training is performed to obtain the multi-class object detection model. This training method integrates classifiers, pseudo-labels, and knowledge distillation. It eliminates the need for manual annotation and adjustment of the existing dataset and fully utilizes all dataset information through knowledge distillation for model training, achieving the goal of simultaneously improving training efficiency and effectiveness.

[0064] Figure 6 This illustration shows a multi-class target detection method proposed in an embodiment of this application. Exemplarily, the multi-class target detection method includes: S210, the target image is input into the multi-class target detection model obtained by the multi-class target detection model training method described above to perform target detection, and the prediction results of all categories of targets present in the target image are obtained.

[0065] As an example, after obtaining a multi-class object detection model, the target image to be detected can be input, and the model can predict and output all the target categories and their location information. For example, if there are three different kinds of fruit in the target image, such as apples, bananas, and dragon fruit, then the model can obtain the category information of these three fruits and their specific location information in the image.

[0066] The training method for this multi-class object detection model can be found in the relevant descriptions in the above embodiments, and will not be repeated here.

[0067] also, Figure 7 A multi-class object detection model training apparatus according to an embodiment of this application is also shown. Exemplarily, the multi-class object detection model training apparatus includes: The teacher network inference module 110 is used to input training images into different pre-trained teacher networks for target detection and output reference feature maps for different detection stages in each teacher network; wherein, different teacher networks are used to detect different categories of targets; the training images contain multiple target categories and have partial target category annotations.

[0068] The positive and negative sample acquisition module 120 is used to obtain positive and negative sample data of the training image in the corresponding category of object detection based on the reference feature map at different detection stages.

[0069] The student network training module 130 is used to input the training images into the student network for multi-class object detection and output the predicted feature maps at each detection stage.

[0070] The loss calculation module 140 is used to calculate the loss value when performing multi-class target detection through the student network based on the positive and negative sample data, the reference feature map, and the predicted feature map.

[0071] The student network training module 130 is also used to perform backpropagation using the loss value to continue training the student network until a preset training stopping condition is met, thereby obtaining a multi-class object detection model.

[0072] It is understood that the apparatus of this embodiment corresponds to the method of the above embodiments, and the options of the method in the above embodiments are also applicable to this embodiment, so they will not be described again here.

[0073] This application also provides a terminal device, which may include, but is not limited to, a robot, a camera device with multi-category target recognition and detection functions, etc. Furthermore, if it is a robot, its specific shape is not limited. Exemplarily, the terminal device includes a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to enable the terminal device to perform the aforementioned multi-category target detection model training method or multi-category target detection method.

[0074] The processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including at least one of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Network Processor (NP), Digital Signal Processor (DSP), Application-Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0075] The memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory is used to store computer programs, and the processor can execute the computer programs accordingly after receiving execution instructions.

[0076] This application also provides a readable storage medium for storing the computer program used in the aforementioned terminal device, wherein, when the computer program is executed on a processor, it implements the multi-class target detection model training method or the multi-class target detection method of the above embodiments.

[0077] The aforementioned readable storage medium can be either a non-volatile storage medium or a volatile storage medium. For example, the readable storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0078] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that, in alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0079] In addition, the functional modules or units in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0080] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a smartphone, personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0081] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for training a multi-class object detection model, characterized in that, include: The training images are input into different pre-trained teacher networks for object detection, and reference feature maps for different detection stages of each teacher network are output. The different teacher networks are used to detect different types of objects. The training images contain multiple object categories and have partial object category labels. The different detection stages include three stages: regression detection stage, object determination stage, and object category determination stage. Based on the reference feature map of each detection stage, the positive and negative sample data of the training image in the corresponding category of object detection are obtained; The training images are input into the student network for multi-class object detection, and the predicted feature maps at each detection stage are output. Based on the positive and negative sample data, the reference feature map, and the predicted feature map, the loss value for multi-class target detection through the student network is calculated, including: using the positive and negative sample data of each detection stage, calculating the knowledge distillation loss value between the reference feature map and the predicted feature map for each detection stage; weighting the knowledge distillation loss values ​​of all detection stages to obtain the total knowledge distillation loss value; calculating the target detection loss value between the predicted feature map and the ground truth feature map obtained based on the target category annotations contained in the training images; and using the total knowledge distillation loss value and the target detection loss value, calculating the loss value for multi-class target detection through the student network. The loss value is used for backpropagation to continue training the student network until a preset training stopping condition is met, thus obtaining a multi-class object detection model.

2. The multi-class target detection model training method according to claim 1, characterized in that, The output includes reference feature maps for different detection stages in each teacher's network, including: Each teacher network outputs a reference feature map at different detection stages according to a preset size. Each point in each reference feature map contains a preset number of anchor boxes, which are used to obtain the positive and negative sample data through parsing.

3. The multi-class target detection model training method according to claim 2, characterized in that, The reference feature maps based on different detection stages are used to obtain positive and negative sample data of the training image in the corresponding category of object detection, including: During the target detection process of the corresponding category, the anchor boxes at the corresponding positions in the reference feature map in each detection stage are assigned as positive or negative samples according to the preset allocation rules. The distinguished positive and negative samples are described in matrix form to obtain the positive and negative sample data of the training image in the corresponding category of object detection.

4. The multi-class target detection model training method according to any one of claims 1 to 3, characterized in that, Each teacher network outputs only a reference feature map for one target category during the target category determination phase; The student network outputs corresponding predicted feature maps for all target category determinations during the target category determination phase.

5. The multi-class target detection model training method according to claim 1, characterized in that, The number of teachers in the network does not exceed the total number of target categories that need to be detected; The different teacher networks were obtained through pre-training, including: Each sample image containing multiple categories of targets is divided into several datasets with different target category labels according to different partial target annotations; Using different datasets, several constructed neural networks are trained for different target detection tasks to obtain several teacher networks for detecting different types of targets, with the same number of datasets as the datasets.

6. A multi-class target detection method, characterized in that, include: The target image is input into a multi-class target detection model obtained by the training method according to any one of claims 1 to 5 to perform target detection, and the prediction results of all categories of targets present in the target image are obtained.

7. A multi-class target detection model training device, characterized in that, include: The teacher network inference module is used to input training images into different pre-trained teacher networks for object detection and output reference feature maps for different detection stages in each teacher network. Different teacher networks are used to detect different categories of objects. The training images contain multiple object categories and have partial object category labels. The different detection stages include three stages: regression detection stage, object determination stage, and object category determination stage. The positive and negative sample acquisition module is used to obtain positive and negative sample data of the training image in the corresponding category of object detection based on the reference feature map of each detection stage; The student network training module is used to input the training images into the student network for multi-class target detection and output the predicted feature maps at each detection stage. The loss calculation module is used to calculate the loss value when performing multi-class target detection through the student network based on the positive and negative sample data, the reference feature map, and the predicted feature map. This includes: calculating the knowledge distillation loss value between the reference feature map and the predicted feature map for each detection stage using the positive and negative sample data of each detection stage; weighting the knowledge distillation loss values ​​of all detection stages to obtain a total knowledge distillation loss value; calculating the target detection loss value between the predicted feature map and the ground truth feature map obtained based on the target category annotations contained in the training images; and calculating the loss value when performing multi-class target detection through the student network using the total knowledge distillation loss value and the target detection loss value. The student network training module is also used to perform backpropagation using the loss value to continue training the student network until a preset training stopping condition is met, thereby obtaining a multi-class object detection model.

8. A terminal device, characterized in that, The terminal device includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the multi-class target detection model training method according to any one of claims 1-5 or the multi-class target detection method according to claim 6.

9. A readable storage medium, characterized in that, It stores a computer program, which, when executed on a processor, implements the multi-class target detection model training method according to any one of claims 1-5 or the multi-class target detection method according to claim 6.

Images